Skip to content. | Skip to navigation

Personal tools


You are here: Home / Wiki / Tutorial


Emulab Tutorial

Emulab Tutorial

Getting Started

This section of the tutorial describes how to run your first Emulab experiment. We cover basic NS syntax and various operational issues that you will need to know in order conduct experiments to completion. Later sections of the tutorial will cover more advanced topics such as loading your own RPMs automatically, running programs automatically, running batch jobs, creating your own disk images and loading those images on your nodes. For your convenience, there is also a Java GUI interface you can use to create NS files for your experiments.

Getting Help

Join the Emulab Users Forum to get help from others in the growing Emulab community.

Logging Into the Web Interface

If you already have an account on Emulab or Netbed, all you need to do is go to Emulab Home Page, enter your login name and your password, and then click on the "Login" button. If you don't have an account, click on the "Join Project" or "Start Project" links. For an overview of how you go about getting an Emulab account, go to the How To Get Started page.

Designing a Network Topology

Part of Emulab's power lies in its ability to assume many different topologies; the description of a such a topology is a necessary part of an experiment. (Note: You may want to take a look at our Java GUI to build experiments without directly editing NS files.) Emulab uses the "NS" ("Network Simulator") format to describe network topologies. This is substantially the same Tcl-based format used by ns-2. Since Emulab offers emulation, rather than simulation, these files are interpreted in a somewhat different manner than ns-2. Therefore, some ns-2 functionality may work differently than you expect, or may not be implemented at all. Please look for warnings of the form:

  	*** WARNING: Unsupported NS Statement!
  	    Link type BAZ, using DropTail!

If you feel there is useful functionality missing, please let us know. Also, some testbed-specific syntax has been added, which with the inclusion of compatibility module (tb_compat.tcl), will be ignored by the NS simulator. This allows the same NS file to work on both Emulab and ns-2, most of the time.

For those unfamiliar with the NS format, here is a small example (We urge all new Emulab users to begin with a small 3-5 node experiment such as this, so that you will become familiar with NS syntax and the practical aspects of Emulab operation). Let's say we are trying to create a test network which looks like the following:


(A is connected to B, and B to C and D with a LAN.)

An NS file which would describe such a topology is as follows. First off, all NS files start with a simple prologue, declaring a simulator and including a file that allow you to use the special tb- commands:

  	# This is a simple ns script. Comments start with #.
  	set ns [new Simulator]
        source tb_compat.tcl

Then define the 4 nodes in the topology.

  	set nodeA [$ns node]
  	set nodeB [$ns node]
  	set nodeC [$ns node]
  	set nodeD [$ns node]

nodeA and so on are the virtual names (vnames) of the nodes in your topology. When your experiment is swapped in, they will be assigned to physical node names like "pc45", probably different ones each time. (NOTE: It's a bad idea to pick vnames that clash with the physical node names in the testbed.)

Next define the link and the LAN that connect the nodes. NS syntax permits you to specify the bandwidth, latency, and queue type. Note that since NS can't impose artificial losses like Emulab can, there is a separate command to add loss on a link. For our example, we will define a full speed LAN between B, C, and D, and a shaped link from node A to B.

  	set link0 [$ns duplex-link $nodeB $nodeA 30Mb 50ms DropTail]
  	tb-set-link-loss $link0 0.01
  	set lan0 [$ns make-lan "$nodeD $nodeC $nodeB " 100Mb 0ms]

In addition to the standard NS syntax above, a number of extensions have been added that allow you to better control your experiment.

For example, you may specify what Operating System is booted on your nodes. We currently support FreeBSD 4.10, FreeBSD 6.2, RedHat Linux 9.0, Fedora Core 4, Fedora Core 6, and Windows XP. Windows 2000 is not supported. Fedora Core 2 (Linux 2.6) is available but unsupported. We also support OSKit kernels on the Emulab PCs.

Click List ImageIDs and OSIDs in the Emulab web interface "Interaction" pane to see the current list of Emulab-supplied OS. By default, our most recent Linux image is selected.

  	tb-set-node-os $nodeA FBSD-STD
  	tb-set-node-os $nodeC RHL-STD
  	tb-set-node-os $nodeC WINXP-UPDATE

In a topology like this, you will likely want to communicate between all the nodes, including nodes that aren't directly connected, like A and C. In order for that to happen, we must enable routing in our experiment, so B can route packets for the other nodes. The typical way to do this is with Static routing. (Other options are detailed below, in the Routing section below.

  	$ns rtproto Static

Lastly, all NS files end with an epilogue that instructs the simulator to start.

 	# Go!
  	$ns run

If you would like to try the above example, the completed NS file can be run as an experiment in your project. Because NS is based on TCL, the full power of the TCL language is available for you to use in your NS files, including loops, control structures, and even procedures/functions. Here's an example of a simple loop: (Download this example: loop.ns)

  	# This is a simple ns script that demonstrates loops.
  	set ns [new Simulator]
  	source tb_compat.tcl
  	set maxnodes 3
  	set lanstr ""
  	for {set i 1} {$i <= $maxnodes} {incr i} {
  	    set node($i) [$ns node]
  	    append lanstr "$node($i) "
  	    tb-set-node-os $node($i) FBSD-STD
  	# Put all the nodes in a lan
  	set big-lan [$ns make-lan "$lanstr" 100Mb 0ms]
  	# Go!
  	$ns run

Beginning the Experiment

After logging on to the Emulab Web Interface, choose the "Begin Experiment" option from the menu. First select which project you want the experiment to be configured in. Most people will be a member of just one project, and will not have a choice. If you are a member of multiple projects, be sure to select the correct project from the menu. Next fill in the Name' and Description' fields. The Name should be a single word (no spaces) identifier, while the Description is a multi word description of your experiment. In the "Your NS file" field, place the local path of an NS file which you have created to describe your network topology. This file will be uploaded through your browser when you choose "Submit." After submission, Emulab will begin processing your request. This will likely take several minutes, depending on how large your topology is, and what other features (such as delay nodes and bandwidth limits) you are using. Assuming all goes well, you will receive an email message indicating success or failure, and if successful, a listing of the nodes and IP address that were allocated to your experiment. For the NS file described above, you would receive a listing that looks similar to this:

  	Virtual Node Info:
  	ID              Type         OS              Qualified Name
  	--------------- ------------ --------------- --------------------
  	nodeA           pc
  	nodeB           pc                 
  	nodeC           pc                 
  	nodeD           pc                 
  	Physical Node Mapping:
  	ID              Type         OS              Physical
  	--------------- ------------ --------------- ------------
  	tbsdelay0       pc850        FBSD47-STD      pc61
  	nodeB           pc850        RHL73-STD       pc63
  	nodeC           pc600        RHL73-STD       pc31
  	nodeD           pc600        RHL73-STD       pc34
  	nodeA           pc600        RHL73-STD       pc13
  	Virtual Lan/Link Info:
  	ID              Member          IP/Mask         Delay     BW (Kbs)  Loss Rate
  	--------------- --------------- --------------- --------- --------- ---------
  	lan0            nodeC:0        0.00      100000    0.000
  	                         0.00      100000    0.000
  	lan0            nodeB:1        0.00      100000    0.000
  	                         0.00      100000    0.000
  	lan0            nodeD:0        0.00      100000    0.000
  	                         0.00      100000    0.000
  	link0           nodeB:0        25.00     30000     0.005
  	                         25.00     30000     0.005
  	link0           nodeA:0        25.00     30000     0.005
  	                         25.00     30000     0.005
  	Physical Lan/Link Info:
  	ID              Member          Delay Node   Delay    BW (Kbs) PLR    Pipe
  	--------------- --------------- ------------ -------- -------- ------ ---------
  	link0           nodeA           tbsdelay0    50.00    30000    0.010  100
  	link0           nodeB           tbsdelay0    50.00    30000    0.010  110
  	Route List:
  	Node            Interface       Dest            Nexthop         Type  Cost
  	--------------- --------------- --------------- --------------- ----- ----
  	nodeA      host  0
  	nodeA      net   0
  	nodeC      net   0
  	nodeD      net   0

A few points should be noted:

  • A single delay node was allocated and inserted into the link between nodeA and nodeB. This link is invisible from your perspective, except for the fact that it adds latency, error, or reduced bandwidth. However, the information for the delay links are included so that you can modify the delay parameters after the experiment has been created.
  • Delays of less than 2ms (per trip) are too small to be accurately modeled at this time, and will be silently ignored. A delay of 0ms can be used to indicate that you do not want added delay; the two interfaces will be "directly" connected to each other.
  • Each link in the "Virtual Lan/Link" section has its delay, etc., split between two entries. One is for traffic coming into the link from the node, and the other is for traffic leaving the link to the node. In the case of links, the four entries often get optimized to two entries in the "Physical Lan/Link" section.
  • The names in the "Qualified Name" column refer to the control network interfaces for each of your allocated nodes. These names are added to the Emulab nameserver map on the fly, and are immediately available for you to use so that you do not have to worry about the actual physical node names that were chosen. In the names listed above, myproj' is the name of the project that you chose to work in, and myexp' is the name of the experiment that you provided in the "Begin an Experiment" page.
  • Please don't use the "Qualified Name" from within nodes in your experiment, since it will contact them over the control network, bypassing the link shaping we configured. (See also these two FAQ entries: here and here.)

Using your Nodes

By the time you receive the email message listing your nodes, the Emulab configuration system will have ensured that your nodes are fully configured and ready to use. If you have selected one of the Emulab-supported operating system images (FreeBSD, Linux, or Windows XP), this configuration process includes:

  • loading fresh disk images so that each node is in a known clean state;
  • rebooting each node so that it is running the OS specified in the NS script;
  • configuring each of the network interfaces so that each one is "up" and talking to its virtual LAN (VLAN);
  • creating user accounts for each of the project members;
  • mounting the projects NFS directory in /proj so that project files are easily shared amongst all the nodes in the experiment;
  • creating a /etc/hosts file on each node so that you may refer to the experimental interfaces of other nodes by name instead of IP number;
  • configuring all of the delay parameters;
  • configuring the serial console lines so that project members may access the console ports from or directly from their desktop.

As this point you may log into any of the nodes in your experiment. You will need to use Secure Shell (ssh), and you should use the `qualified name' from the nodes mapping table so that you do not form dependencies on any particular physical node. Your login name and password will be the same as your Web Interface login and password. The /etc/hosts file on each node will provide a local name mapping for the other nodes in your experiments. You should take care to use these names (or IP numbers) and not the names listed in the node mapping, since the Emulab names refer to the control network LAN that is shared amongst all nodes in all experiments. It is only the experimental interfaces that are entirely private to your experiment. (See also these two FAQ entries: here and here.) NOTE: The configuration process just described occurs only on Emulab constructed operating system images. If you are using an OSKit kernel, or your own disk image with your own operating system, you will be responsible for all of the configuration. At some point we hope to provide tools to assist in the configuration, but for now you are on your own. Ask Testbed Ops for help or more information.

I need root access!

If you need to customize the configuration, or perhaps reboot nodes, you can use the "sudo" command, located in /usr/local/bin on FreeBSD and Linux, and /usr/pkg/bin on NetBSD. All users are added to the Administrators group on Windows XP nodes. Our policy is very liberal; you can customize the configuration in any way you like, provided it does not violate Emulab's administrative policies. As as example, to reboot a node that is running FreeBSD:

  	/usr/local/bin/sudo reboot


My node is wedged!

This is bound to happen when running experimental software and/or experimental operating systems. Fortunately we have an easy way for you to power cycle nodes without requiring Tested Operations to get involved. If you must power cycle a node, log on to and use the "node_reboot" command:

  	node_reboot <node> [node ... ]

where node is the physical name, as listed in the node mapping table. You may provide more than one node on the command line. Be aware that you may power cycle only nodes in projects that you are member of. Also, node_reboot does its very best to perform a clean reboot before resorting to cycling the power to the node. This is to prevent the damage that can occur from constant power cycling over a long period of time. For this reason, node_reboot may delay a minute or two if it detects that the machine is still responsive to network transmission. In any event, please try to reboot your nodes first (see above). You may also reboot all the nodes in an experiment by using the -e option to specify the project and experiment names. For example:

  	node_reboot -e testbed,multicast

will reboot all of the nodes reserved in the "multicast" experiment in the "testbed" project. This option is provided as a shorthand method for rebooting large groups of nodes.

I've scrogged my disk!

Scrogging your disk is certainly not as common, but it does happen. You can either terminate your experiment, and recreate it (which will allocate another group of nodes), or if you prefer you can reload the disk image yourself. You will of course lose anything you have stored on that disk; it is a good idea to store only data that can be easily recreated, or else store it in your project directory in /proj. Reloading your disk with a fresh copy of the default image is easy, and requires no intervention by Emulab staff:

  	os_load <node> [node ... ]

os_load will wait (not exit) until the nodes have been reloaded, so that you do not need to check the console lines of each node to determine when the load is done.

I've finished my experiment

When your experiment is completed, and you no longer need the resources that have been allocated to it, you will need to terminate the experiment via the Emulab Web Interface. Click on the "End An Experiment" link. You will be presented with a list of all of the experiments in all of the projects for which you have the authorization to terminate experiments. Select the experiment you want to terminate by clicking on the button in the "Terminate" column on the right hand side. You will be asked to confirm your choice. The Emulab configuration system will then tear down your experiment, and send you an email message when the process is complete. At this point you are allowed to reuse the experiment name (say, if you wanted to create a similar experiment with different parameters).

Scheduling experiment swapout/termination

If you expect that your experiment should run for a set period of time, but you will not be around to terminate or swap the experiment out, then you should use the scheduled swapout/termination feature. This allows you to specify a maximum running time in your NS file so that you will not hold scarce resources when you are offline. This can also be used in conjunction with 'batch mode' experiments to ensure that they do not run longer than is reasonably expected. To schedule a swapout or termination in your NS file:

     $ns at 2000.0 "$ns terminate"
     $ns at 2000.0 "$ns swapout"

This will cause your experiment to either be terminated or swapped out after 2000 seconds of wallclock time.

Getting Help!

If you have any questions or problems, or just want to comment on Emulab's operation (maybe you want to suggest an improvement to one of the Web pages), feel free to contact us by sending email to Testbed Operations. Please send all correspondence to this address, not to individual members of our team. Messages that aren't on this list may not get responses. Also note that much of the software is in development, and occasionally things might break or not work as you expect. Again, please feel free to contact us.

Advanced Topics

A more advanced example

We have a more advanced example demonstrating the use of RED queues, traffic generators, the event system, and program objects.

The Emulab "Control Network"

The Emulab control network is the source of a number of problems related to the behavior of network links in an experiment. So in this section we attempt to explain exactly what the control network is, why problems occur, and how to avoid them. Every physical node in the testbed has one interface connected to a common 100Mb LAN. This interface is used by the testbed infrastructure to configure experiments (e.g. distribute account info, load disks, etc.). It is also used by experimenter to communicate with the nodes from outside Emulab or from (e.g., ssh). Finally, it may be used by an experimenter to monitor activity during an experiment. Control net links have a fixed address and not configured by the user. The control network is differentiated from the experimental network which is the set of links specified in the experiment topology over which experiment applications should communicate. Experiment links are configured by the user and may be shaped and otherwise controlled by that user. To illustrate the difference, consider the simple two-nodes-and-a-router configuration described by the NS snippet:

  	source tb_compat.tcl
  	set ns [new Simulator]
  	set node1 [$ns node]
  	set router [$ns node]
  	set node2 [$ns node]
  	set linkA [$ns duplex-link $node1 $router 100Mb 0ms DropTail]
  	set linkB [$ns duplex-link $router $node2 1Mb 10ms DropTail]
  	$ns rtproto Static
  	$ns run

which is instantiated as shown in red in Figure 1.


Figure 1. Control network interfaces (blue) to an experiment topology (red).

From your vantage point out in the Internet, or on the machine "," you can access a node in your experiment via its canonical Emulab name (e.g., "") or by the DNS alias that is assigned when your experiment is created (e.g., ""). Both types of names use the fixed 100Mb control network link (IP: to access the node. These control net links are shown in blue in Figure 1. The view from inside an experiment node is different. In an ideal world, applications running inside the experiment would not even know about the control network, since it is not part of the topology. However, the control net must be visible to nodes for several practical reasons, most notably allowing login from remote sites and the ability of the node to access the shared NFS /proj and /users filesystems. As the control net is visible to applications on a node, it can lead to its inadvertent use by applications. In the above example, consider a ping from node1 to node2. The expectation is that traffic will pass through the included router and over the shaped link to node2, resulting in round-trip times of 20ms:

    1> ping node2
    PING node2-linkB ( from : 56(84) bytes of data.
    64 bytes from node2-linkB ( icmp_seq=1 ttl=63 time=20.3 ms
    64 bytes from node2-linkB ( icmp_seq=2 ttl=63 time=20.3 ms

But if the ping travels over the control network rather than the experimental network, there will be no delay:

     2 ping
     PING ( from : 56(84) bytes of data.
     64 bytes from ( icmp_seq=1 ttl=64 time=0.291 ms
     64 bytes from ( icmp_seq=2 ttl=64 time=0.124 ms

Notice the subtle difference between the pings, the correct one uses an unqualified name, the incorrect one uses the fully qualified name. Unqualified names are resolved from a local /etc/hosts file that is created on each node. Fully qualified names are resolved by the Emulab nameserver and return the routable, control net address. Figure 1 shows the various names each node can be named by, and which interface they will resolve to. Accidental use of the control net interface in an experiment commonly occurs due to one of three reasons:

  • The experiment user configures an application incorrectly. By specifying either a node's fully qualified name or its control net address when configuring an application, that application will use the wrong interface.
  • An application itself decides which interface to use based on a node's hostname. Since the hostname is set to the fully qualified name, that name will resolve to the control net address.
  • An application uses all available interfaces. By default, many server applications will listen on all interfaces they discover via ioctls or other kernel mechanisms. This will include the control interface.

In the first case, you just need to be careful to use the correct name or address. For the latter two cases, you might need to modify the application. Fortunately, most applications include options enabling you to explicitly specify which interfaces to use (or not use). We recognize that this is not ideal, and will be trying to find better ways to "hide" the control net in the future.

Installing RPMS automatically

The Emulab NS extension tb-set-node-rpms allows you to specify a (space separated) list of RPMs to install on each of your nodes when it boots:

  tb-set-node-rpms $nodeA /proj/myproj/rpms/silly-freebsd.rpm
  tb-set-node-rpms $nodeB /proj/myproj/rpms/silly-linux.rpm
  tb-set-node-rpms $nodeC /proj/myproj/rpms/silly-windows.rpm

The above NS code says to install the silly-freebsd.rpm file on nodeA, the silly-linux.rpm on nodeB, and the silly-windows.rpm on nodeC. RPMs are installed as root, and must reside in either the project's /proj directory, or if the experiment has been created in a subgroup, in the /groups directory. You may not place your rpms in your home directory.

Installing TAR files automatically

The Emulab NS extension `tb-set-node-tarfiles` allows you to specify a set of tarfiles to install on each of your nodes when it boots. While similar to the `tb-set-node-rpms` command, the format of this command is slightly different in that you must specify a directory in which to unpack the tar file. This avoids problems with having to specify absolute pathnames in your tarfile, which many modern tar programs balk at.

  tb-set-node-tarfiles $nodeA /usr/site /proj/myproj/tarfiles/silly.tar.gz

The above NS code says to install the silly.tar.gz tar file on nodeA from the working directory /usr/site when the node first boots. The tarfile must reside in either the project's /proj directory, or if the experiment has been created in a subgroup, in the /groups directory. You may not place your tarfiles in your home directory. You may specify as many tarfiles as you wish, as long as each one is preceded by the directory it should be unpacked in, all separated by spaces.

Starting your application automatically

You can start your application automatically when your nodes boot for the first time (experiment is started or swapped in) by using the tb-set-node-startcmd NS extension. The argument is a command string (pathname of a script or program, plus arguments) that is run as the UID of the experiment creator, after the node has reached multiuser mode. The command is invoked using /bin/csh, and the working directory is undefined (your script should cd to the directory you need). You can specify the same program for each node, or a different program. For example:

  tb-set-node-startcmd $nodeA "/proj/myproj/runme.nodeA"
  tb-set-node-startcmd $nodeB "/proj/myproj/runme.nodeB"

will run /proj/myproj/runme.nodeA on nodeA and /proj/myproj/runme.nodeB on nodeB. The programs must reside on the node's local filesystem, or in a directory that can be reached via NFS. This is either the project's /proj directory, in the /groups directory if the experiment has been created in a subgroup, or a project member's home directory in /users. If you need to see the output of your command, be sure to redirect the output into a file. You can place the file on the local node, or in one of the NFS mounted directories mentioned above. For example:

     tb-set-node-startcmd $nodeB "/proj/myproj/runme >& /tmp/foo.log"

The exit value of the start command is reported back to the Web Interface, and is made available to you via the "Experiment Information" link. There is a listing for all of the nodes in the experiment, and the exit value is recorded in this listing. The special symbol none indicates that the node is still running the start command. The start command is implemented using Program Objects, which are described in more detail in the Advanced Tutorial. Note that the start command is run only when the experiment is swapped in. If you want to rerun the experiment, you can swap the experiment out and back in, or you can reboot all of the nodes in your experiment. If rebooting, you must fire off the program object(s) yourself by restarting the Event System on

  	eventsys_control <proj> <expt> replay

You can also control each program object by sending it events, either with the NS "at" command:

    $ns at 2000.0 "$nodeA_startcmd stop"
    $ns at 2010.0 "$nodeA_startcmd start"

or you can use the event program on

    tevc -e myproj/myexpt now nodeA_startcmd stop
    tevc -e myproj/myexpt now nodeA_startcmd start

The start command is especially useful when combined with 'batch mode' experiments.

How do I know when all my nodes are ready?

It is often necessary for your start program to determine when all of the other nodes in the experiment have started, and are ready to proceed. Sometimes called a barrier, this allows programs to wait at a specific point, and then all proceed at once. Emulab provides a simple form of this mechanism using a synchronization server that runs on a node of your choice. You specify the node in your NS file:

    tb-set-sync-server $nodeB

When nodeB boots, the synchronization server will automatically start. Your software can then synchronize using the emulab-sync program that is installed on your nodes. For example, your node start command might look like this:

   if [ "$1" = "master" ]; then
       /usr/testbed/bin/emulab-sync -i 4
       /usr/testbed/bin/emulab-sync fi /usr/site/bin/dosilly

In this example, there are five nodes in the experiment, one of which must be configured to operate as the master, initializing the barrier to the number of clients (four in the above example) that are expected to rendezvous at the barrier. The master will by default wait for all of the clients to reach the barrier. Each client of the barrier also waits until all of the clients have reached the barrier (and of course, until the master initializes the barrier to the proper count). Any number of clients may be specified (any subset of nodes in your experiment can wait). If the master does not need to wait for the clients, you may use the async option which releases the master immediately:

    /usr/testbed/bin/emulab-sync -a -i 4

You may also specify the name of the barrier.

    /usr/testbed/bin/emulab-sync -a -i 4 -n mybarrier /usr/testbed/bin/emulab-sync -n mybarrier

This allows multiple barriers to be in use at the same time. Scripts on nodeA and nodeB can be waiting on a barrier named "foo" while (other) scripts on nodeA and nodeC can be waiting on a barrier named "bar." You may reuse an existing barrier (including the default barrier) once it has been released (all clients arrived and woken up).

Setting up IP routing between nodes

As Emulab strives to make all aspects of the network controllable by the user, we do not attempt to impose any IP routing architecture or protocol by default. However, many users are more interested in end-to-end aspects and don't want to be bothered with setting up routes. For those users we provide an option to automatically set up routes on nodes which run one of our provided FreeBSD, Linux or Windows XP disk images. You can use the NS rtproto syntax in your NS file to enable routing:

    $ns rtproto ''protocol''

where the protocol option is limited to one of Session, Static, Static-old, or Manual.

Session routing provides fully automated routing support, and is implemented by enabling gated running the OSPF protocol on all nodes in the experiment. This is not supported on Windows XP nodes. Static routing also provides automatic routing support, but rather than computing the routes dynamically, the routes are precomputed by a distributed route computation algorithm running in parallel on the experiment nodes. Static-old specifies use of the older centralized route computation algorithm, precomputing the nodes when the experiment is created, and then loading them onto each node when it boots. Manual routing allows you to explicitly specify per-node routing information in the NS file. To do this, use the Manual routing option to rtproto, followed by a list of routes using the add-route command:

    $node add-route $dst $nexthop

where the dst can be either a node, a link, or a LAN. For example:

    $client add-route $server $router
    $client add-route [$ns link $server $router] $router
    $client add-route $serverlan $router

Note that you would need a separate add-route command to establish a route for the reverse direction; thus allowing you to specify differing forward and reverse routes if so desired. These statements are converted into appropriate route(8) commands on your experimental nodes when they boot. In the above examples, the first form says to set up a manual route between $client and $server, using $router as the nexthop; $client and $router should be directly connected, and the interface on $server should be unambiguous; either directly connected to the router, or an edge node that has just a single interface.


If the destination has multiple interfaces configured, and it is not connected directly to the nexthop, the interface that you are intending to route to is ambiguous. In the topology shown to the right, $nodeD has two interfaces configured. If you attempted to set up a route like this:

    $nodeA add-route $nodeD $nodeB

you would receive an error since it cannot be determined (easily, with little programmer effort, by Emulab staff!) which of the two links on $nodeD you are referring to. Fortunately, there is an easy solution, courtesy of an Emulab extension. Instead of a node, specify the link directly:

    $nodeA add-route [$ns link $nodeD $nodeC] $nodeB

This tells us exactly which link you mean, enabling us to convert that information into a proper route command on $nodeA. The last form of add-route command is used when adding a route to an entire LAN. It would be tedious and error prone to specify a route to each node in a LAN by hand. Instead, just route to the entire network:

    set clientlan [$ns make-lan "$nodeE $nodeF $nodeG" 100Mb 0ms]
    $nodeA add-route $clientlan $nodeB

While all this manual routing infrastructure sounds really nifty, its probably a good idea to use either Session or Static routing for all but small, simple topologies. Explicitly setting up all the routes in even a moderately-sized experiment is extremely error prone. Consider this: a recently created experiment with 17 nodes and 10 subnets required 140 hand-created routes in the NS file. Yow! Two final, cautionary notes on routing:

  • You might be tempted to set the default route on your nodes to reduce the number of explicit routes used. Don't do it. That would prevent nodes from contacting the outside world, i.e., you. The default route must be set to use the control network interface.
  • If you use your own routing daemon, you must avoid using the control network interface in the configuration. Since every node in the testbed is directly connected to the control network LAN, a naive routing daemon configuration will discover that any node is just one hop away, via the control network, from any other node and all inter-node traffic will be routed via that interface.

Batch Mode

Batch Mode Introduction

Batch Mode experiments can be created on Emulab via the "Create an Experiment" link in the operations menu to your left. There is a checkbox near the bottom of the form that indicates you want to use the batch system. There are several important differences between a regular experiment and a batch mode experiment:

  • The experiment is run when enough resources (ie: nodes) are available. This might be immediately, or it might be sometime in the future.
  • Once your NS file is handed off to the system, the batch system is responsible for setting up the experiment and tearing it down once the experiment has completed. You will receive email notifying you when the experiment has been scheduled and when it has been terminated.
  • Your NS file must define a start command to run on each node using the `tb-set-node-startcmd` NS extension. It is the exit value(s) of the start command(s) that indicates that the experiment is completed; when all of the nodes have run their respective start commands and reported their exit values, the batch system will then tear down the experiment.

The batch system may also be used for "queuing" interactive experiments. By leaving out tb-set-node-startcmd commands in the NS file, an experiment will be started when enough nodes are available, but then remain active until you (or the system) swap or terminate it. Existing Batch Mode experiments (both true batch experiments and queued interactive ones) may be "stopped" and "queued." Stopping a Batch Mode experiment has identical semantics to swapping out a regular experiment, including loss of local node disk state, and thus is not really a "clean stop." Queueing a Batch Mode experiment is almost the same as a regular experiment swapin, the difference being that the swapin may be delayed if insufficient resources are available at the time of the swapin.. Unfortunately, while Batch Mode makes running experiments much easier, it also makes wasting resources far easier, e.g., a user queues up an interactive experiment and then is not around when it finally gets swapped in. To counter this, the idle swap threshold for a Batch Mode experiment is typically set lower than that for a regular experiment, usually 15 minutes. If necessary, this can be changed using the "Edit Experiment Metadata" link on an experiment's page. The modified value will remain in effect til the next swapout. While multiple Batch Mode experiments may be running in Emulab simultaneously, the current policy is that there can only be one such experiment per-user, per-project. That is, two users in the same project may run experiments simultaneously and one user may run simultaneous experiments in different projects. Finally, note that the rights and responsibilities of a Batch Mode experiment are conferred at creation time. It is not possible to, for example, swap out a regular experiment and then swap it back in as a Batch Mode experiment (i.e., to get delayed swapin semantics). This is unfortunate.

A Batch Mode Example

Consider example NS file batch.ns. First off, we have to arrange for the experimental software to be automatically installed when the nodes boot. This is done with the `tb-set-node-tarfiles` NS extension:

    tb-set-node-tarfiles $nodeA /usr/site /proj/testbed/tarfiles/silly.tar.gz
    tb-set-node-tarfiles $nodeB /usr/site /proj/testbed/tarfiles/silly.tar.gz

The next two lines of the NS file specify what program should be run on each of the nodes. Using the tb-set-node-startcmd NS extension, we specify the name of the program to run once all the nodes have booted and are ready to proceed:

    tb-set-node-startcmd $nodeA "/usr/site/bin/run-silly >& /tmp/foo.log"
    tb-set-node-startcmd $nodeB "/usr/site/bin/run-silly >& /tmp/foo.log"

After you have been notified via email that the batch experiment is running, you can track the progress of your experiment by looking in the "Experiment Information" page. As each node completes the start command, the listing for that node will be updated to reflect the exit status of the command (you may need to hit the Reload button to see the changes). Once all of the nodes hare reported in an exit status, the batch system will tear down the experiment and send you email. If your experiment is such that one node is the controller, and runs commands on all the other nodes, then simply run a dummy start command on the other nodes so that the batch system will receive an exit value for that node. Since the batch is not terminated until all nodes have reported in, be sure that the controlling node does not exit from its start command until all of the nodes have finished. A dummy start command can be setup like this:

    tb-set-node-startcmd $nodeC "/bin/echo"

The status of your batch experiment can be viewed via the "Experiment Information" link in the Web Interface Options menu. You may also cancel a batch after you have submitted it using the "Terminate" option in the information display. You may also stop a batch job, causing it to swap out by using the "Stop" option. The batch may be reposted at any time. The batch system is still under development. Currently, the batch system tries every 10 minutes to run your batch. It will send you email every 5 or so attempts to let you know that it is trying, but that resources are not available. It is a good idea to glance at the message to make sure that the problem is lack of resources and not an error in your NS file.

Custom OS Images

If your set of operating system customizations cannot be easily contained within an RPM/TAR (or multiple RPM/TARs), then you can create your own custom OS image; Emulab allows you to create your own disk images and load them on your experimental nodes, automatically when your experiment is created or swapped in. Once you have created a custom disk image, you can use it in your NS file. When your experiment is swapped in, the testbed system will arrange for your disks to be loaded. Our experience has shown that it is much faster to load a disk image on 10 nodes at once, then it is to load a bunch of RPMS or tarballs on each node as it boots. So, while it may seem like overkill to create your own disk image, we can assure you it often is not!

The most common approach (described in more detail below) is to start with a single a single node that is running one of Emulab's standard Linux or FreeBSD operating systems, customize that node as needed, and then use the web interface to create a new image. You can then use that image in subsequent experiments by specifying the descriptor name in your NS file with the `tb-set-node-os` directive. When the experiment is configured, the proper image will be loaded on each node automatically by the Emulab system.

Creating your Custom Image

As mentioned above, a typical approach to creating your own disk image using one of the default images as a base, goes like this:

  1. Create a single-node Linux or FreeBSD experiment. In your NS file, use the appropriate tb-set-node-os directive. For example, one of the following statements:
               tb-set-node-os $nodeA FBSD90-STD
               tb-set-node-os $nodeA FEDORA15-STD
               tb-set-node-os $nodeA UBUNTU12-64-STD
    The precise list of available default images is listed in the Emulab web interface, under the Experimentation drop down menu, using the List Imageids option.
  2. After your experiment has swapped in (you have received the email saying it is running), log into the node and load all of the RPMS and software packages that you wish to load (if you want to install the latest version of the Linux kernel on one of our standard RedHat/Fedora disk images, or on your own custom Linux image, see the Emulab FAQ. Be sure to arrange for any programs that need to be started at boot time. It is a good idea to reboot the node and make sure that everything is running as you want it to when it comes up.
  3. If you are creating a Windows based image, you must "prepare" the node. This is discussed in great detail in the Windows page.
  4. Note the physical (pcXX) name of the machine used.
  5. Go to the Emulab web interface, and navigate to the experiment information page for your experiment. The quickest way to get there is to click on My Emulab at the top of the page, then on the Experiments tab. Find your experiment and click on that.
  6. Click on the node you have customized.
  7. Click on the Create a Disk Image link in the menu on the left hand side.
  8. Your are now looking at a form to provide all the information needed to create the new image. Most of the info is already filled in, since you have started with a default image. You must however, fill in the Descriptor Name which is the ImageID by which your new image will be known (how you will use it in your NS file).
  9. Do you need to check the Whole Disk Image box? By default, only a single partition is captured. If you added another slice (partition) or ran mkextrafs, and you want that data in your custom image, then check the box. If you do not want that data, then edit /etc/fstab and remove the entry. Otherwise your new image probably will not boot.
  10. Click the submit button.
  11. Wait for the email saying the image creation is done. This can take a while, so get a cup of coffee or take a walk around the block (several times).

The next step is to test your image in a new experiment to make sure you have not introduced any dependencies on the original experiment, and to make sure the image was properly created by the Emulab. Similar to above, start another single node experiment and be sure to use specify the ImageID in your NS file:

	tb-set-node-os $nodeA YourImageID

Submit your experiment and wait for to swap in. Log into the second node and make sure everything is running as you want it to. If all is well, terminate both of your experiments.

If something has gone wrong, terminate the second experiment, go back to the first experiment and fix things up as necessary on the node. Next you need to update your custom image via the Emulab web interface as described in the next section. Test your image again with your second experiment, and repeat until done.

Updating your Custom Image

Once you have your image, it is easy to update it later by taking a new snapshot from a node running your image. Assuming you have swapped in an experiment with a node running your image, and you have made changes to that node, use the Emulab web interface to navigate to the descriptor page for your image:

  1. Use the Experimentation drop down menu, and choose List Imageids to see the entire list of Images to which you have access to.
  2. Find your custom image and click on it.
  3. In the More Options menu, click on "Snapshot Node ..."
  4. Fill in the name of the node that is running your image, and click on "Go"
  5. As in the above instructions, wait for the email saying your image has been updated before you try and use the image.


Deleting your Custom Image

Many Emulab sites enable disk quotas, which means you cannot store an unlimited number of custom images. Eventually you will need to delete some of them. This is easily accomplished; navigate to the descriptor page for your image:

  1. Use the Experimentation drop down menu, and choose List Imageids to see the entire list of Images to which you have access to.
  2. Find your custom image and click on it.
  3. Now click on Delete this Image Descriptor under More Options.
  4. You will now see a Confirmation page. Note the checkbox to retain the image file (the big bucket of bits). Normally you want to leave this box checked so that both the descriptor and the file are deleted. But sometimes you want to keep the image file, say if you want to recreate the descriptor for some reason.
  5. Click on the Confirm button.