Ixp

Integration of Intel IXP network processors

The IXP1200 network processor is now integrated into Emulab, and available to external researchers for remote experimentation. Similar to other Emulab resources, an IXP1200 node can be allocated through an ns script or through the Netbuild GUI. It can be configured as part of an arbitrary topology (of PCs, IXPs, or other node types) with end-nodes, traffic generators, routers, links etc. Through NFS it has access to /proj and your Emulab home directory. You can configure a command to run automatically upon bootup. You have full access to the serial consoles of both the IXP and its host PC, which you can power cycle and of course reboot.

All that makes it easy to allocate, experiment with, and evaluate an IXP-based application.

Currently, we have installed 8 ENP2505 cards (aka Bridalveil), with Intel SDK 2.01 software. We hope to incorporate more IXP1200 nodes as we make space in our machine room, and also IXP2400 cards as resources permit.

Getting Started in Emulab

To start using IXP nodes in Emulab, you first need an account in Emulab. See Emulab Documentation for more details; especially if you are using Emulab for the first time.

Creating an IXP1200 experiment

An IXP1200 node can be allocated in an experiment in two ways: using an ns script or through a Java GUI called Netbuild.

Using ns file
Using Netbuild

Using ns file

Here's a sample ns file, which allocates an IXP1200 node and two end-nodes connected in a simple linear topology.

  # Boilerplate
  source tb_compat.tcl
  set ns [new Simulator]
  # Create an ixp node
  set myixp [$ns node]
  # Set its hardware type to ixp-bv (bridalveil)
  tb-set-hardware $myixp ixp-bv
  # XXX temporary: explicitly name the IXP host, bind it to the IXP and
  # set its OS.  (We should automatically set the OS so these steps would
  # not be required.)
  set ixphost [$ns node]
  tb-bind-parent $myixp $ixphost
  tb-set-node-os $ixphost RHL73-IXPHOST
  # Create end nodes
  set n0 [$ns node]
  set n1 [$ns node]
  # Create links between the end nodes and the IXP
  $ns duplex-link $n0 $myixp 100Mb 0ms DropTail
  $ns duplex-link $myixp $n1 100Mb 0ms DropTail
  # Set up static routing on all nodes,
  $ns rtproto Static
  # Boilerplate
  $ns run

When the above ns file is submitted, Emulab creates an experiment with the requested nodes, creates the topology specified above, and does the following:

Routes are automatically generated between all nodes; interface configuration information like IP addresses is generated for the IXP interfaces as well as other nodes. Emulab uses this to create config files for the Intel L3Fwdr reference design (route.l3fwdr, ixsys.l3fwdr) as a sample. The user can run this application to get familiar with the MicroAce architecture, and also to understand how the IXPs are set up in Emulab.
The IXP card control interface (pci/eth interface) is set up, and the card is reachable/can reach the outside world through the host.
An experimenter's project directory (/proj/$projectid) and the SDK 2.01 workspace (/opt) are mounted on the card.
Hosts file, dns resolver files are automatically set up.
The experimenter can login to the card and run the l3fwdr application as a test application to see how things are set up. To login to the card, ssh into the IXP host. You can then either telnet to the card or access the card console from the IXP host. The linux on the card has just a root account with no password.
To telnet to the card, run
```
        telnet -l root myixp.myexp.myproj.emulab.net
```
where myixp is the name of your ixp card in your ns file, myexp is your experiment name and myproj is your project name.
To access the card's console, do
```
        sudo minicom ixp
```
Once logged into the card, do the following:
```
        cd /opt/ixasdk/bin/arm-be
        ./ixstart /opt/config/ixsys.l3fwdr
```
This will start up the reference L3Fwdr application on the card. You can then ping the end-nodes through the IXP router, or ping the end-nodes from the IXP.

Using Netbuild

You can use Netbuild , a Java GUI, for creating your experiment topology. See Netbuild Documentation for details. When setting the type of the IXP node, use ixp-bv.

More complicated topologies with LANs, emulated links, routers etc., can be similarly created to test your IXP based application, using either an ns file or Netbuild.

Running your custom application on the IXP1200

Typically, an experimenter will write his/her own application, and evaluate it inside Emulab. Once you have done some basic testing of your application, you can automate its startup in your experiment. To do this, Emulab supports startup commands which are run on the card on bootup.

Startup Commands
A sample Startup Command

Startup Commands

Your ns script can include this:

    tb-set-node-startup $myixp "/opt/ixasdk/bin/arm-be/mycustomapp /opt/ixasdk/bin/arm-be/mycustomapp.config"

This will start up the above command line on the IXP1200 card after the card has booted up. To create a config file for your custom application, you might need access to interface/route configuration that Emulab has generated. This information is stored by Emulab in a file whose path is exported on the card in the environment variable $USERCONFIG. The $USERCONFIG file has interface/route configuration lines with the following format:

    # interface number IP Netmask Broadcast Mac
    interface 0 10.1.2.2 10.1.2.255 255.255.255.0 00:00:AC:11:AC:00
    # route destination netmask gateway interface-number route 10.1.2.2 255.255.255.0 0.0.0.0 0

This format is deliberatly similar to the format used by MicroAce config files. This information can be used by your startup command to generate config files on the fly and then invoke your custom application with the generated config files. For example, consider an application which does routing (using the Intel L3Fwdr reference design) and adds its own functionality like Filtering. Your startup command can use the emulab config file to generate the route config files, then invoke your application with these config files. Your application (for example ixconfig) in turn will configure all the MicroAces, download code to the microengines etc.

A sample Startup Command

Emulab creates route config files which are exported on the card in /opt/config/ixsys.l3fwdr and /opt/config/route.l3fwdr. You can try the following startup command in an experiment.

     tb-set-node-startup $myixp "/opt/ixasdk/bin/arm-be/ixstart /opt/config/ixsys.l3fwdr"

This will initialize and download Intel's L3Fwdr Reference design to the microengines.

Test Experiment to get started

Before creating your own topologies, its advisable to try the following first, to get familiar with IXP configuration inside Emulab.

Use the above sample ns file to create an experiment. Log into the IXP node, then manually start the L3fwdr application.
```
    cd /opt/ixasdk/bin/arm-be
    ./ixstart /opt/config/ixsys.l3fwdr
```

Test the application by using ping between the end-nodes.

Add a startup command for the IXP node. Once an experiment is already created, you can still add a startup command by editing a node's information. From your experiment webpage, click on the ixp node in the Node ID column. Then click "Edit Node Info" in the Node Options table. Add the following startup command in the "Startup Command" entry in the form.
```
    /opt/ixasdk/bin/arm-be/ixstart /opt/config/ixsys.l3fwdr
```

Then reboot the IXP host, this time you do not have to manually start the routing application, it is started automatically by Emulab. Again try pinging between end-nodes to see if the application works.

Once you are familiar with how the IXP1200 is set up inside Emulab, you could try richer topologies and your own startup commands.

Node Usage Policies

Emulab has certain Node Usage Policies to facilitate sharing of resources among experimenters. Since we have limited number of boards available, please use resources in accordance with these policies.

Setup Notes

Two filespaces are mounted on the card (on the linux on the StrongArm): your project directories are mounted on the card at /proj/$projectid, and the Intel SDK 2.01 is mounted at /opt. Files that you create in /proj/$projectid survive across experiments and swapouts of the same experiment. Files in /opt are lost across experiments and swapouts. So typically, you would want to save your code (MicroAces, apps etc) in /proj/$projectid. Look at the following FAQ entry, Where do I store files needed by my experiment? for details.
Currently, Windows developer workbench is not supported. So you would have to compile your microcode on your windows machine, then copy it over to the card. You cannot use the windows based debugger. However printf's equivalent method of debugging works fine; your microcode can write debug information to Scratch memory and an application on the StrongArm can read and print it. This is a useful debugging technique.

Internals of IXP Support

Nothing detailed, but the general issues and design are outlined in some Powerpoint slides.