LinuxHappy's Weblog

Failures and Successes of Linux-based Development

Fun: C API for Expect in an hour

Posted by linuxhappy on January 15, 2008

Ok, I’ve got about an hour or two before I have to sleep, so the goal of this blog is to get familiar with the C API for Expect ( before I go zZzzzz. Lets get something straight:
1. I’ve never used tcl/tk (nor do I know what it is really capable of)
2. I’ve never used expect
3. This exercise is purely for just getting myself familiar with this C API, perhaps dig into tcl/tk, and perhaps dig into expect a little bit… Ultimately learning *something* new.
4. I’m not a http protocol expert
5. After googling around I noticed that I couldn’t find too many articles about using the C API for expect…. So here’s my attempt.

I’m developing on my trusty Ubuntu 7.10 laptop which is working out just spiffy… I decided it was finally time to get the gcc man pages… Crediting the following page ( I got all the doc/man pages/info pages I needed:

sudo apt-get install gcc-doc manpages-dev manpages-posix-dev

Now I can do silly things like `man 3 write` to answer my silly questions like, “oh wait, what does write return again?”

Ok, so first, we gotta get Tcl and expect. I guess I don’t need to install tk because I don’t plan on controlling anything X related.. Ok, so after a couple of tries, I came up with the following:

#Set up a working directory and download files
mkdir ~/expect
cd ~/expect
tar -xzvf expect.tar.gz
tar -xzvf tcl8.4.17-src.tar.gz

#Compiling tcl
cd tcl8.4.17/unix
./configure --prefix=~/expect/tcl-install --enable-shared
make install
make install-private-headers
cd ../../

#Compiling expect
cd expect-5.43
#Note: use your own home dir ;)
./configure --prefix=/home/jwoo/expect/tcl-install/ --with-tcl=/home/jwoo/expect/tcl-install/lib \
  --with-tclinclude=/home/jwoo/expect/tcl-install/include/  --enable-shared
make install

If that doesn’t work, then maybe I’ve got something on my build system that you don’t, but it is pretty straightforward on how to compile and install. Also you might have noticed that I installed tcl and expect in my home directory. I tend to do this because I rather not mix nice and neat ubuntu/debian packages with these source installed tar.gz’s. Now it’s time to link your own custom user space application against the expect library and the tcl library! In this case I have decided to create a simple telnet session that simply requests for an (html) webpage, and dumps it to a screen. hooray! haha.

mkdir ~/expect/htmlrip && cd ~/expect/htmlrip
touch ~/expect/htmlrip/htmlrip.c
touch ~/expect/htmlrip/Makefile

Inside my makefile I have the following:


CFLAGS=-I../tcl-install/include -L../tcl-install/lib -lexpect5.43 -ltclstub8.4
        gcc  -o htmlrip htmlrip.c $(CFLAGS)

        rm htmlrip -f

Inside my C file, I’ve got a request, to request


#include <stdio.h>
#include <string.h>

#include "expect.h"
#define APPNAME "htmlrip"
int main()
  int fd;
  int i=0;
  int retval=0;
  //Thank you wireshark ;)
  char getstr[] = "GET / HTTP/1.1\r\nHost:\r\nUser-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/20071204 Ubuntu/7.10 (gutsy) Firefox/\r\nAccept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip,deflate\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nKeep-Alive: 300\r\nConnection: keep-alive\r\n";

  unsigned char rcv_buf[4096]__attribute__((aligned(4)));

  printf("Starting htmlrip\n");
  fd = exp_spawnl("telnet", "telnet", "", "http", (char *)0);
  retval = write(fd, getstr, strlen(getstr));
  if (retval < 0)
      printf("%s: Failed to write to telnet session: %d\n", APPNAME, retval);
      return retval;
    printf("%s: Wrote %d bytes to fd=%d\n", APPNAME, retval, fd);

  printf("Printing out html\n");
  while (retval !=-1)
      memset(rcv_buf, 0, sizeof(rcv_buf));
      retval = read(fd, rcv_buf, sizeof(rcv_buf));
      for(i =0; i < retval; i++)
          printf("%c", rcv_buf[i]);
  return 0;

Finally to Build/Compile/Run:

cd ~/expect/htmlrip/
LD_LIBRARY_PATH=../tcl-install/lib ./htmlrip

Ok ok. here’s some warning: when you run it this program, it’ll request the google webpage, and at the end, it’ll say: “Your client has issued a malformed or illegal request”… I’m pretty certain I screwed up the HTTP request some how, because the same http request works for simple html pages. Maybe someone can fill me in on what I’m doing wrong… but.. the focus of this project was to explore the expect C api (not the http protocol.)


One Response to “Fun: C API for Expect in an hour”

  1. If you work on OSX (like me), you’d probably want to upgrade your Xcode installation (to >=3.2.1) and compile tcl like he did for /unix, not necessarily for /macosx since ld (linker) patches aren’t available through installing the Iphone SDK which also includes Xcode.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: