Personal tools
You are here: Home / Wiki / Xen


Xen-based Emulab virtual nodes

Xen-based Emulab virtual nodes

Getting started

Emulab vnodes based on Xen virtual machines is now the standard. OpenVZ-based and FreeBSD jail-based vnodes are no longer supported.

The executive summary is that you specify a Xen vnode by setting a node in your topology to be type "pcvm" or "nodetype-vm" (e.g., "d710-vm") and then choosing a supported OSID for the vnode. Each vnode will have a minimum 1024MB of RAM and a 6-16 GB virtual disk. These are arbitrary constants that can be overridden to some degree; see below. Shared filesystems are available in the vnodes. Vnodes can be connected with shaped virtual links, though expect lower fidelity emulations than with physical links. Each vnode has a control net connection, but they have unroutable IP addresses so you cannot communicate to/from vnodes and the outside (of Emulab) world.

NOTE: until this page has been fully updated, you should look through the description of Emulab's OpenVZ virtual nodes since many of the concepts and details are very similar and apply directly to XEN based virtual nodes. Go ahead, I'll wait here till you come back ...


Here is an example NS file for a three node, two link topology (node0 <=> node2 <=> node1) with multiple OSes:

set ns [new Simulator]
source tb_compat.tcl

# Nodes
set node0 [$ns node]
set node1 [$ns node]
set node2 [$ns node]

# Set the link multiplexing technology. Do not change this.
# Needed for experiments with multiple vnode hosts.
tb-set-vlink-emulation "vlan"

# Choose the vnode OSes.
# Most standard Emulab images should work.
tb-set-node-os $node0 UBUNTU14-64-STD
tb-set-node-os $node1 CENTOS63-64-STD
tb-set-node-os $node2 FBSD102-64-STD

# Set the type of the nodes to reflect they are vnodes.
# Use <nodetype>-vm (e.g., "d710-vm") to force a particular HW type.
tb-set-hardware $node0 pcvm
tb-set-hardware $node1 pcvm
tb-set-hardware $node2 pcvm

## Set non-fatal swapin flag.
## Useful for debugging when a vnode fails to boot and you want to investigate.
#tb-set-node-failure-action $node0 "nonfatal"
#tb-set-node-failure-action $node1 "nonfatal"
#tb-set-node-failure-action $node2 "nonfatal"

## Override the default amount of RAM given to a vnode.
## The default is 1024 (1GB).
#tb-set-node-memory-size $node0 2048
#tb-set-node-memory-size $node1 2048
#tb-set-node-memory-size $node2 2048

# Links
set link0 [$ns duplex-link $node0 $node2 100000.0kb 0.0ms DropTail]
set link1 [$ns duplex-link $node1 $node2 100000.0kb 0.0ms DropTail]

$ns rtproto Static
$ns run

Guest OSs

It is best that you explicitly select an OSID to use for your guest OS. There is a default, XEN-STD, which is literally a copy of the host dom0 OS currently 64-bit Ubuntu 14.04 LTS), but it is slow to setup and its identity may change over time. You may use one of a number of our standard images (that run on bare metal). The requirement is that either the kernel be compiled with XEN PV support or that its is a 64-bit image. As of this time, the supported standard images are:

  • UBUNTU14-64-STD: 64-bit Ubuntu 14.04 LTS, 3.13.0 kernel.
  • UBUNTU12-64-STD: 64-bit Ubuntu 12.04 LTS, 3.2.46 kernel.
  • CENTOS63-64-STD: 64-bit CentOS 6.3, 3.x.x kernel.
  • FBSD102-64-STD: 64-bit FreeBSD 10.2.
  • FBSD93-64-STD: 64-bit FreeBSD 9.3.
  • FEDORA15-STD: 32-bit Fedora 15.
  • FBSD92-STD: 32-bit FreeBSD 9.2.

All of these images are available on the Utah Emulab, but should also run on whatever Emulab you are using; ask your local administrator to import images you require from Utah's download directory.

You can also create a customize image from any of these images and use that on a Xen vnode. Moreover, you can customize that image while running on a vnode and use the resulting image on bare metal!

Controlling memory

As mentioned above, you can control the amount of memory each guest receives:

tb-set-node-memory-size $node0 2048

The units is always MB, and the resource mapper will tell you if you asked for too much or you have exhausted the amount on memory available on the node. This is especially important on Emulab's shared nodes, where you are competing with other experiments for resources on the same physical host.

Controlling disk space

Each XEN guest is given enough disk space to hold the requested image. Most Emulab images are built within a 6GB or 16GB partition, with about 25-50% of the disk space used by the operating system files. If the remaining space is not enough for your needs, you can request additional disk space with the following NS statement:

$mynode  add-attribute XEN_EXTRAFS 5

The units is always GB, and at this time we enforce a hard limit of 10GB.  As with Emulab physical nodes, the extra disk space will appear in the fourth slice of your guest's disk. You can turn this extra space into a useable file system by logging into your guest and doing:

mynode> sudo mkdir /dirname
mynode> sudo /usr/local/etc/emulab/ /dirname

Creating a custom image

Emulab allows you to create a Custom OS Image of a virtual node, much like you are able to create a custom image to run on a physical node. Your custom image is saved and loaded in much the same way as described for physical nodes in the tutorial.

The difference is in how you create the image after you have setup your virtual node the way you want it. Once you are ready, go to the Show Experiment page for your experiment, and click on the node you want to save. One of the menu options is "Create a Disk Image". Click on that link and follow the instructions. If you are customizing one of the Emulab provided images for the first time, you will need to complete the form and click submit. If you updating your own image, then you just need to click on the confirmation button.

For all practical concerns, a disk image created from a XEN node may also be run on a bare metal node. You might need to edit the image descriptor in the web interface, to add the required node types, but in general images are interchangeable. You may also click on the whole disk image on the web form, which will create an image that includes your extra file system (see above).


Link shaping of virtual links is done using tc running in dom0, which is a variant of Emulab's end node shaping. Note that you cannot dynamically modify link shaping parameters.

Controlling Xen VM's

If you are using Emulab's shared nodes, you can ignore this section; users are not able to log into the physical host (dom0).

When dom0 has booted up you can ssh directly to it and use either Emulab commands or Xen commands to control the domU's. These need to be run with root priveleges (use sudo). Some helpful Emulab commands:

    # Halt all vnodes
    /usr/local/etc/emulab/bootvnodes -h

    # Boot all vnodes
    /usr/local/etc/emulab/bootvnodes -b

    # Kill all vnodes (destroy virtual disks after halting)
    /usr/local/etc/emulab/bootvnodes -k

Some helpful Xen commands:

    # Start a domU
    /usr/sbin/xl create <some-config-file>

    # Halt a vm (clean shutdown)
    /usr/sbin/xl shutdown <vm name>

    # Kill a vm (unclean shutdown)
    /usr/sbin/xl destroy <vm name>

    # List all vm's
    /usr/sbin/xl list

    # Connect to the console of a vm
    /usr/sbin/xl console <vm name>

Modifying Xen VM instances

Xen VM's use a configuration file to specify various attributes about their setup. Once dom0 has booted you can modify these configuration files as you like. The configuration files live in /var/emulab/vms/<vm-name>/xm.conf. Typical things to modify are the kernel and memory size. It is possible to modify the disk sizes as well but you have to use LVM commands directly and usually you have to create all the disks at once.

Any changes will not be persisted across reboots of dom0 (or use of the bootvnodes command above), so save your changes somewhere if you need them.

Windows on Xen

In theory, yes. In practice, not yet.


No doubt.

Implementation details and other info

Building xen from source