Emulab Tutorial - A More Advanced Example
Here is a slightly more complex example demonstrating the use of RED queues, traffic generation, and the event system. Where possible, we adhere to the syntax and operational model of ns-2, as described in the NS manual.
- RED/GRED Queues: In addition to normal DropTail links, Emulab supports the specification of the RED and GRED (Gentle RED) links in your NS file. RED/GRED queuing is handled via the insertion of a traffic shaping delay node, in much the same way that bandwidth, delay, and packet loss is handled. For a better understanding of how we support traffic shaping, see the ipfw and dummynet man pages on users.emulab.net. It is important to note that Emulab supports a smaller set of tunable parameters then NS does; please read the aforementioned manual pages!
- Traffic Generation: Emulab supports Constant Bit Rate (CBR) traffic generation, in conjunction with either Agent/UDP or Agent/TCP agents. We currently use the TG Tool Set to generate traffic.
- Event System: Emulab supports limited use of the NS at syntax, allowing you to define a static set of events in your NS file, to be delivered to agents running on your nodes. There is also "dynamic events" support that can be used to inject events into the system on the fly, say from a script running on users.emulab.net.
- Program Objects: Emulab has added extensions that allow you to run arbitrary programs on your nodes, starting and stopping them at any point during your experiment run.
- Link Tracing and Monitoring: Emulab supports simplified tracing and monitoring of links and lans.
What follows is an sample NS file that demonstrates the above features, with annotations where appropriate. First we define the 2 nodes in the topology:
set nodeA [$ns node] set nodeB [$ns node]
Next define a duplex link between nodes nodeA and nodeB. Instead of a standard DropTail link, it is declared to be a Random Early Detection (RED) link. While this is obviously contrived, it allows us to ignore [tutorial.php3#Routing routing] issues within this example.
set link0 [$ns duplex-link $nodeA $nodeB 100Mb 0ms RED]
Each link is has an NS "Queue" object associated with it, which you can modify to suit your needs (currently, there are two queue objects per duplex link; one for each direction. You need to set the parameters for both directions, which means you can set the parameters asymmetrically if you want). The following parameters can be changed, and are defined in the NS manual (see Section 7.3).
set queue0 [[$ns link $nodeA $nodeB] queue] $queue0 set gentle_ 0 $queue0 set red_ 0 $queue0 set queue-in-bytes_ 0 $queue0 set limit_ 50 $queue0 set maxthresh_ 15 $queue0 set thresh_ 5 $queue0 set linterm_ 10 $queue0 set q_weight_ 0.002
The maximum limit_ for a queue is 1 megabyte if it is specified in bytes, or 100 slots if it is specified in slots (each slot is 1500 bytes).
In the case of a LAN, there is a single queue object for every node that is a member of the LAN and it refers to the node-to-lan direction. The only special case is a 100Mb 0ms LAN that does not use end node shaping. No queue object is available in that case. Here is an example that illustrates how to get handles on the queue objects of a LAN so as to change the parameters:
set n0 [$ns node] set n1 [$ns node] set n2 [$ns node] set lan0 [$ns make-lan "$n0 $n1 $n2" 100Mb 0ms] set q0 [[$ns lanlink $lan0 $n0] queue] set q1 [[$ns lanlink $lan0 $n1] queue] set q2 [[$ns lanlink $lan0 $n2] queue] $q0 set limit_ 20 ...
A UDP agent is created and attached to nodeA, then a CBR traffic generator application is created, and attached to the UDP agent:
set udp0 [new Agent/UDP] $ns attach-agent $nodeA $udp0 set cbr0 [new Application/Traffic/CBR] $cbr0 set packetSize_ 500 $cbr0 set interval_ 0.005 $cbr0 attach-agent $udp0
A TCP agent is created and also attached to nodeA, then a second CBR traffic generator application is created, and attached to the TCP agent:
set tcp0 [new Agent/TCP] $ns attach-agent $nodeA $tcp0 set cbr1 [new Application/Traffic/CBR] $cbr1 set packetSize_ 500 $cbr1 set interval_ 0.005 $cbr1 attach-agent $tcp0
You must define traffic sinks for each of the traffic generators created above. The sinks are attached to nodeB:
set null0 [new Agent/Null] $ns attach-agent $nodeB $null0 set null1 [new Agent/TCPSink] $ns attach-agent $nodeB $null1
Then you must connect the traffic generators on nodeA to the traffic sinks on nodeB:
$ns connect $udp0 $null0 $ns connect $tcp0 $null1
Lastly, a set of events to control your applications and link characteristics:
$ns at 60.0 "$cbr0 start" $ns at 70.0 "$link0 bandwidth 10Mb duplex" $ns at 80.0 "$link0 delay 10ms" $ns at 90.0 "$link0 plr 0.05" $ns at 100.0 "$link0 down" $ns at 110.0 "$link0 up" $ns at 115.0 "$cbr0 stop" $ns at 120.0 "$cbr1 start" $ns at 130.0 "$cbr1 set packetSize_ 512" $ns at 130.0 "$cbr1 set interval_ 0.01" $ns at 140.0 "$link0 down" $ns at 150.0 "$cbr1 stop"
When you receive email containing the experiment setup information (as described in Beginning an Experiment), you will notice an additional section that gives a summary of the events that will be delivered during your experiment:
Event Summary: -------------- Event count: 18 First event: 60.000 seconds Last event: 160.000 seconds
You can get a full listing of the events for your experiment by checking the 'Details' pane on the 'Show Experiment' page for your experiment. This report will include a section like this:
Event List: Time Node Agent Type Event Arguments ------------ ------------ ---------- ---------- ---------- ------------ 60.000 nodeA cbr0 TRAFGEN START PACKETSIZE=500 RATE=100000 INTERVAL=0.005 70.000 tbsdelay0 link0 LINK MODIFY BANDWIDTH=10000 80.000 tbsdelay0 link0 LINK MODIFY DELAY=10ms 90.000 tbsdelay0 link0 LINK MODIFY PLR=0.05 100.000 tbsdelay0 link0 LINK DOWN 110.000 tbsdelay0 link0 LINK UP 115.000 nodeA cbr0 TRAFGEN STOP 120.000 nodeA cbr1 TRAFGEN START PACKETSIZE=500 RATE=100000 INTERVAL=0.005 130.000 nodeA cbr1 TRAFGEN MODIFY PACKETSIZE=512 130.000 nodeA cbr1 TRAFGEN MODIFY INTERVAL=0.01 140.000 tbsdelay0 link0 LINK DOWN 150.000 tbsdelay0 link0 LINK UP 160.000 nodeA cbr1 TRAFGEN STOP
The above list represents the set of events for your experiments, and are stored in the Emulab Database. When your experiment is swapped in, an event scheduler is started that will process the list, and send them at the time offset specified. In order to make sure that all of the nodes are actually rebooted and ready, time does not start ticking until all of the nodes have reported to the event system that they are ready. At present, events are restricted to system level agents (Emulab traffic generators and delay nodes), but in the future we expect to provide an API that will allow experimenters to write their own event agents.
Dynamic Scheduling of Events
NS scripts give you the ability to schedule events dynamically; an NS script is just a TCL program and the argument to the "at" command is any valid TCL expression. This gives you great flexibility in a simulated world, but alas, this cannot be supported in a practical manner in the real world. Instead, we provide a way for you to inject events into the system dynamically, but leave it up to you to script those events in whatever manner you are most comfortable with, be it a PERL script, or a shell script, or even another TCL script! Dynamic event injection is accomplished via the Testbed Event Client (tevc), which is installed on your experimental nodes and on users.emulab.net. The command line syntax for tevc is:
tevc -e proj/expt time objname event [args ...]
where the time parameter is one of:
- +seconds (floating point or integer)
For example, you could issue this sequence of events.
tevc -e testbed/myexp now cbr0 set interval_=0.2 tevc -e testbed/myexp +10 cbr0 start tevc -e testbed/myexp +15 link0 down tevc -e testbed/myexp +17 link0 up tevc -e testbed/myexp +20 cbr0 stop
Some points worth mentioning:
- There is no "global" clock; Emulab nodes are kept in sync with NTP, which does a very good job of keeping all of the clocks within 1ms of each other.
- The times "now" and "+seconds" are relative to the time at which each event is submitted, not to each other or the start of the experiment.
- The set of events you can send is currently limited to control of traffic generators and delay nodes. We expect to add more agents in the future.
- Sending dynamic events that intermix with statically scheduled events can result in unpredictable behavior if you are not careful.
- Currently, the event list is replayed each time the experiment is swapped in. This is almost certainly not the behavior people expect; we plan to change that very soon.
- tevc does not provide any feedback; if you specify an object (say, cbr78 or link45) that is not a valid object in your experiment, the event is silently thrown away. Further, if you specify an operation or parameter that is not appropriate (say, "link0 start" instead of "link0 up"), the event is silently dropped. We expect to add error feedback in the future.
This is a (mostly) comprehensive list of events that you can specify, either in your NS file or as a dynamic event on the command line. In the listings below, the use of "link0", "cbr0", etc. are included to clarify the syntax; the actual object names will depend on your NS file. Also note that when sending events from the command line with tevc, you should not include the dollar ($) sign. For example:
|NS File:||$ns at 3.0 "$link0 down"|
|tevc:||tevc -e proj/expt +3.0 link0 down|
In "ns" script: $link0 bandwidth 10Mb duplex $link0 delay 10ms $link0 plr 0.05 With "tevc": tevc ... link0 modify bandwidth=20000 # In kbits/second; 20000 = 20Mbps tevc ... link0 modify delay=10ms # In msecs (the "ms" is ignoredd) tevc ... link0 modify plr=0.1 Both: $link0 up $link0 down
- Queues: Queues are special. In your NS file you modify the actual queue, while on the command line you use the link to which the queue belongs.
$queue0 set queue-in-bytes_ 0 $queue0 set limit_ 75 $queue0 set maxthresh_ 20 $queue0 set thresh_ 7 $queue0 set linterm_ 11 $queue0 set q_weight_ 0.004
- CBR: interval_ and rate_ are two way of specifying the same thing. iptos_ allows you to set the IP_TOS socket option for a traffic stream.
$cbr0 start $cbr0 set packetSize_ 512 $cbr0 set interval_ 0.01 $cbr0 set rate_ 10Mb $cbr0 set iptos_ 16 $cbr0 stop
Event Groups allow you to conveniently send events to groups of like objects. For example, if you want to bring down a set of links at the same time, you could do it one event at a time:
$ns at 100.0 "$link0 down" $ns at 100.0 "$link1 down" $ns at 100.0 "$link2 down" $ns at 100.0 "$link3 down"
which works, but is somewhat verbose. Its also presents a problem when sending dynamic events with tevc from the shell:
tevc -e proj/expt now link0 down tevc -e proj/expt now link1 down tevc -e proj/expt now link2 down tevc -e proj/expt now link3 down
These four events will be separated by many milliseconds as each call to tevc requires forking a command from the shell, contacting boss, sending the event to the event scheduler, etc.
A better alternative is to create an event group, which will schedule events for all of the members of the group, sending them at the same time from the event scheduler. The example above can be more simply implemented as:
set mylinks [new EventGroup $ns] $mylinks add $link0 $link1 $link2 $link3 $ns at 60.0 "$mylinks down"
From the command line:
tevc -e proj/expt now mylinks down
- All of the members of an event group must be of the same type; you cannot, say, put a link and a program object into the same event group since they respond to entirely different commands. The parser will reject such groups.
- An object (such as a link or lan) can be in multiple event groups.
- Event groups are not hierarchical; you cannot put one event group into another event group. If you need this functionality, then you need to put the objects themselves (such as a link or lan) into each event group directly:
set mylinks1 [new EventGroup $ns] set mylinks2 [new EventGroup $ns] $mylinks1 add $link0 $link1 $link2 $link3 $mylinks2 add $link0 $link1 $link2 $link3
We have added some extensions that allow you to use NS's at syntax to invoke arbitrary commands on your experimental nodes. Once you define a program object and initialize its command line and the node on which the command should be run, you can schedule the command to be started and stopped with NS at statements. To define a program object:
set prog0 [$nodeA program-agent -command "/bin/ls -lt"] set prog1 [$nodeB program-agent -command "/bin/sleep 60"]
Then in your NS file a set of static events to run these commands:
$ns at 10 "$prog0 start" $ns at 20 "$prog1 start" $ns at 30 "$prog1 stop"
If you want to schedule starts and stops using dynamic events:
tevc -e testbed/myexp now prog0 start tevc -e testbed/myexp now prog1 start tevc -e testbed/myexp +20 prog1 stop
If you want to change the command that is run (override the command you specified in your NS file), then:
tevc -e testbed/myexp now prog0 start COMMAND='ls'
Some points worth mentioning:
- A program must be "stopped" before it is started; if the program is currently running on the node, the start event will be silently ignored.
- The command line is passed to /bin/csh; any valid csh expression is allowed, although no syntax checking is done prior to invoking it. If the syntax is bad, the command will fail. It is a good idea to redirect output to a log file so you can track failures.
- The "stop" command is implemented by sending a SIGTERM to the process group leader (the csh process). If the SIGTERM fails, a SIGKILL is sent.
As a special case, you can also run a program on the users node:
set opsprog [new Program $ns] $opsprog set node "ops" $opsprog set command "/bin/ls -lt >& /tmp/foo"
But this is something you should rarely need to do ...
Link Tracing and Monitoring
Emulab has simple support for tracing and monitoring links and lans. For example, to trace a link:
set link0 [$ns duplex-link $nodeB $nodeA 30Mb 50ms DropTail] $link0 trace
The default mode for tracing a link (or a lan) is to capture just the packet headers (first 64 bytes of the packet) and store them to a tcpdump output file. You may also view a realtime summary of packets through the link, via the web interface, by going to the 'Show Experiment' page for the experiment, and clicking on the Link Tracing/Monitoring' menu option. You may also control the link tracing programs on the fly, pausing them, restarting them, and killing them.
Here is an example of the output from interactive 'Link Tracing/Monitoring':
1177100936.506386 (icmp:0,0 tcp:0,0 udp:0,0 other:0,0) (icmp:0,0 tcp:0,0 udp:0,0 other:0,0) (dropped:0)
The first thing on each line is a timestamp, in typically unix format (seconds since the epoch). Each set of measurements in parentheses contains counts for one of the interfaces on the node to which you're connected. When you first connect, it prints a single line containing the number of interfaces for the node, and then one line explaining which link each interface belongs to. The first set of measurements on the line corresponds to the first interface reported, and so on. 'icmp:A,B' means that, since the last message, there were A bytes of ICMP traffic sent in B packets.
In addition to capturing just the packet headers, you may also capture the entire packet:
$link0 trace packet
Or you may not capture any packets, but simply gather the realtime summary so that you can view it via the web interface.
$link0 trace monitor
By default, all packets traversing the link are captured by the tracing agent. If you want to narrow the scope of the packets that are captured, you may supply any valid tcpdump (pcap) style expression:
$link0 trace monitor "icmp or tcp"
You may also set the snaplen for a link or lan, which sets the number of bytes that will be captured by each of the trace agents (as mentioned above, the default is 64 bytes, which is adequate for determining the type of most packets):
$link0 trace_snaplen 128
Tracing parameters may also be specified on a per-node basis, for each node in a link or lan. For example, consider the duplex link link0 above between nodeA and nodeB. If you want to set a different snaplen and trace expression for packets leaving nodeA, then:
[[$ns link $nodeA $nodeB] queue] set trace_type header [[$ns link $nodeA $nodeB] queue] set trace_snaplen 128 [[$ns link $nodeA $nodeB] queue] set trace_expr "ip proto tcp"
To set the parameters for packets leaving nodeB, simply reverse the arguments to the link statement:
[[$ns link $nodeB $nodeA] queue] set trace_snaplen 128
For a lan, the syntax is slightly different. Consider a lan called lan0 with a node called nodeL on it:
[[$ns lanlink $lan0 $nodeL] queue] set trace_type header [[$ns lanlink $lan0 $nodeL] queue] set trace_snaplen 128 [[$ns lanlink $lan0 $nodeL] queue] set trace_expr "ip proto tcp"
When capturing packets (rather then just "monitoring"), the packet data is written to tcpdump (pcap) output files in /local/logs on the delay node. Note that while delay nodes are allocated for each traced link/lan, packets are shaped only if the NS file requested traffic shaping. Otherwise, the delay node acts simply as a bridge for the packets, which provides a place to capture and monitor the packets, without affecting the experiment directly. Whether the link or lan is shaped or not, packets leaving each node are captured twice; once when they arrive at the delay node, and again when they leave the delay node. This allows you to precisely monitor how the delay node affects your packets, whether the link is actually shaped or not. You can use the loghole utility to copy the capture files back to the experiment's log directory.
When a link or lan is traced, you may monitor a realtime summary of the packets being captured, via the web interface. From the "Show Experiment" page for your experiment, click on the "Link Tracing" link. You will see a list of each link and lan that is traced, and each node in the link or lan. For each node, you can click on the "Monitor" button to bring up a window that displays the realtime summary for packets leaving that node. Currently, the realtime summary is somewhat primitive, displaying the number of packets (and total bytes) sent each second, as well as a breakdown of the packet types for some common IP packet types.
Other buttons on the Link Tracing page allow you to temporarily pause packet capture, restart it, or even kill the packet capture processes completely (since they continue to consume CPU even when paused). The "snapshot" button causes the packet capture process to close its output files, rename them, and then open up new output files. The files can then be saved off with the loghole utility, as mentioned above.
If you want to script the control of the packet tracing processes, you can use the event system to send dynamic events. For example, to tell the packet capture processes monitoring link0 to snapshot, pause, and restart:
tevc -e myproj/myexp now link0-tracemon snapshot tevc -e myproj/myexp now link0-tracemon stop tevc -e myproj/myexp now link0-tracemon start
And of course, you may use the NS "at" syntax to schedule static events from your NS file:
$ns at 10 "$link0 trace stop" $ns at 20 "$link0 trace start" $ns at 30 "$link0 trace snapshot"
The output files that the capture process create, are stored in /local/logs, and are named by the link and node name. In the above link example, four capture files are created:
where the .recv files hold the packets that were sent by the node and received by the delay node. The .xmit files hold those packets that were transmitted by the delay node and received by the other side of the link. So, for packets sent from nodeA to nodeB, the packet would arrive at the delay node and be recorded in trace_nodeA-link0.recv. Once the packet traverses the delay node (subject to Dummynet traffic shaping) and it is about to be transmitted, it is recorded in trace_nodeA-link0.xmit. By comparing these two files, you can see how the Dummynet traffic shaping has affected your packets, in each direction. Note that even if you have not specified traffic shaping, you still get the same set of files. In this case, the .recv and .xmit files will be nearly identical, reflecting only the negligible propagation delay through the software bridge.
When you issue a "snapshot" command, the above files are closed, and renamed to:
and a new set of files is opened. Note that the files are not rolled; the next time you issue the snapshot command, the current set of files ending with .0 are lost.
Endnode tracing/monitoring refers to putting the trace hooks on the end nodes of a link, instead of on delay nodes. This happens when there are no delay nodes in use, such as when using multiplexed virtual nodes, or if you have explicitly requested end node shaping to reduce the number of nodes you need for an experiment. You may also explicitly request tracing to be done on the end nodes of a link (or a lan) with the following NS command:
$link0 trace_endnode 1
(Note that if a delay node does exist, it will be used for traffic capture even if endnode tracing is specified.) When tracing/monitoring is done on an endnode, the output files are again stored in /local/logs, and are named by the link and node name. The difference is that there is just a single output file, for those packets leaving the node. Packets are captured after traffic shaping has been applied. Endnode tracing can also be used on Planetlab nodes setup through Emulab's Planetlab portal. In this case, all packets sent or received are recorded in the .xmit file.