What does Emulab install on a "conforming" client OS?
First, it should be said that you should be able to run and use any operating system or other standalone bootable kernel (e.g., an OSKit kernel) that supports the hardware machines and peripheral devices in the testbed. This would be a so-called MINIMAL OS that Emulab will load onto a disk, reboot, and just declare it ready. The user has to do pretty much everything. At a minimum they must have either:
- pre-configured the serial console in the OS so that they can access the machine, or
- pre-configured the proper network device driver and arranged for the OS to DHCP on the correct interface and provide some sort of network login capability.
Once they can talk to the machine, they can then manually configure anything else they want, install software packages, etc.
What we generally call an "Emulab supported" image, is an OS install that has a suite of Emulab scripts installed that automatically take care of tasks like: determining which hardware devices are present and initializing them, setting up user accounts on the machine, configuring network interfaces, mounting shared filesystems, running monitoring and verification tools, starting and stopping network traffic or node applications, etc. Running an OS with these tools installed makes for a more complete and satisfying Emulab Experience.
The OS itself
The first piece is configuring the OS kernel to use the hardware present in the testbed. There are well-established mechanisms for doing this with Unix-like OSes such as FreeBSD or Linux (i.e., dynamic detection of hardware and loading of device drivers). Something similar exists for Windows, but it isn't used much outside OEMs and large corporate installations; i.e., not us.
Ideally, the OS is also configured to redirect its console to the serial port, since most Emulab installations have a serial port connected and exported to users.
There is a make target in the Emulab source tree for installing "the client side" but this assumes that the OS you are installing on is supported. Various scripts and C programs might have to be tweaked to Do The Right Thing on a particular OS. If you have a Unix-like OS, modifying the scripts to work is probably fairly simple. If it is not a Unix-like OS, you may be in for a long slog. Note that Windows XP support is currently provided via the Cygwin environment--there is no "native" support for Windows.
There are, broadly speaking, four classes of tools that are installed and used on a client node:
- "Standard" tools. These are common third-party packages that exist for most open source OSes and are used by the various other programs and scripts.
- Node bootstrap tools. These are scripts and programs that run EVERY TIME a node boots. They are not invoked while nodes are running. These tools largely use the tmcc/tmcd protocol for acquiring information from the control infrastructure.
- Node monitoring tools. These are scripts and programs that run continuously and wakeup periodically (on the order of every few minutes) whenever a node is running as part of an experiment. They report status of nodes to the control infrastructure. These tools use a variety of network protocols to communicate with the control infrastructure.
- Emulab event agents. These are scripts and programs that run continuously, waking up whenever they receive an Emulab event from the control infrastructure. These event-driven tools perform actions on behalf of users: start/stop programs, start/stop/modify network traffic generation, take interfaces up and down. They all use the common "pubsub" event distribution protocol.
- Perl scripting language, version 5.8 or greater. The majority of the boostrap scripts and many of the other tools are written in Perl.
- ISC DHCP client utility, version 3.x. Used to identify and configure the control network interface.
- NFS client, version 2 or 3. For mounting shared filesystems from server.
- OpenSSH, "recent" version (proto 2) For login access to nodes. Root@boss uses this for rebooting node as well.
- OpenSSL, 0.9.8+ Needed for tmcc and the event system.
Needed for optional or less common uses
- Mysql database client, 5.0 or above. If your testbed is configured with the USERDB feature.
- Linux RPM utility. If you want to install software RPM packages.
- GNU tar. If you want to install software "tarball" packages.
- Gated, the last released version. For "session" style (aka, dynamic) routing. Doubtful this can even be found anymore, needs to be replaced, probably by quagga or zebra.
- iperf, rude and crude For linktest. The client build process will download these automatically from the Emulab mothership at Utah.
- tg, version 2.0. For the traffic generator agent. Also auto-downloaded.
- rsync For elabinelab builds.
- TCL, version ? For NSE, the "emulator" part of NS that allows combined emulated and simulated nodes. This has probably severely bit-rotted so you don't want to go here lightly.
For doing a from-source build, you will also need
- GNU autoconf 2.13 or newer. To configure the Emulab source.
- GNU make. To build the Emulab source.
- GCC 3.x or above and standard "C library". To build many of the Emulab tools.
- Lex (flex) and Yacc (bison). To build pubsub and the traffic generator.
- Python scripting language, version 2.2 or greater. Required by Emulab configure (is this actually used on the client?)
- Boost 1.31 or newer To build the shortest-path IP route calculator. I think we just use the headers (templates?) and not the libraries, so I don't think this is needed for running the program.
- Git to pull the sources to your machine
- g++, flex, bison, python, byacc, libtool, libssl, libpcap, libboost, libmysqlclient
There are a number of OS-specific files that need to be modified or installed to get all the Emulab-required standard tools running and to fire off the Emulab-specific configuration scripts. As an example in FreeBSD, variables are set in /etc/rc.conf to handle enabling, disabling, and modification of stardard software tools. The hook we use to fire off the Emulab configuration scripts is /usr/local/etc/rc.d/testbed.sh. The client-side tools and scripts installed by Emulab in a client are described in exquisite and never-ending detail in the attached spreadsheet and in ClientSelfConfig.
So how do I actually make a "conforming" client OS?
No good document exists, because it is a major pain in the ass. The general process is:
- Allocate a machine.
- Install a fresh version of whatever OS you want. This will most likely require someone at Utah physically installing some boot media into the machine.
- Customize the OS as described above.
- Install the Emulab client-side.