Skip to content. | Skip to navigation

Personal tools


You are here: Home / Wiki / NetBooting


Network booting techniques used in Emulab
How things boot: a brief summary of the various techniques we use to
network boot nodes and load MFSes.

0. What we can boot.

   All of our nodes are configured to network (PXE) boot and, whereever
   possible, to NEVER boot from the hard drive. The latter is to ensure
   that if the network boot fails, we won't boot some potentially unknown
   OS from the hard drive and that we will try the network boot again.
   In the olden days, for nodes that we could not prevent from falling
   back to a disk boot, we would install an MBR boot loader that just
   immediately rebooted (well, it would give the user a prompt for a
   couple of seconds and then reboot--just in case they really wanted
   to boot from disk).

   The first level boot loader, loaded via PXE, is typically specified
   via DHCP as the "filename" argument. However, this is not always true
   as we will see below. This "pxeboot" will then talk to Emulab via
   the bootinfo protocol to see what to do next. The options are:

   * Boot from a disk/partition combo.
     The disk number is a BIOS disk number (e.g., 0x80). If no disk is
     specified, then boot from disk 0.

   * Boot from a disk/partition with a kernel command line.
     This was mostly useful for OSKit kernel "back in the day", but is
     also used to select an alternate kernel or select the HZ rate of
     a delay node. The actual semantics of the command line depends on
     the combination of the boot loader and kernel. At the very least
     an alternate kernel name (the first arg) should work. Additional
     command line arguments may or may not make it to the kernel and
     some may be interpreted as environment settings in the loader and
     those may or may not make it to the kernel. In other words, use
     of a command line for anything other than setting the HZ rate of
     a delay node kernel may or may not work!

   * Boot a kernel using a memory-based root filesystem (MFS).
     MFSes are used typically when loading a disk or creating an image
     of a disk. The former, the so-called "frisbee" MFS, is not intended
     to be a general OS, its sole purpose is to run the frisbee disk
     loader. The latter is known as the "admin" MFS because it can be
     used for purposes other than just capturing a disk image; e.g., if
     you have screwed up the disk somehow and want to get on and look at
     or fix it. This MFS is intended to be more general purpose. There
     is a third "newnode" MFS that is used only when a machine is first
     booted, to report info back to Emulab about this new node.

   Two stated transitions are related to the initial boot process:
   PXEBOOTING and BOOTING. The first says that a node has made a PXE
   request, the latter says that the OS is booting. Since the node
   cannot self-report these transitions (as it does all others) we
   have to report them by proxy. Typicially, but not always, the
   bootinfo server on boss is the proxy. Whenever a node makes a
   "what do I do next" request, the server reports both transitions.

1. The "classic" Emulab legacy BIOS boot path.

   The original and still most highly used bootstrap path involves PXE
   booting via the BIOS, a FreeBSD-based boot loader, and FreeBSD-based
   MFSes. In this path, /tftpboot/pxeboot.emu is specified in the dhcpd.conf
   file as the program to download. pxeboot is a PXE-savvy version of
   the FreeBSD "stage 2" boot loader which has been modified to talk
   to the bootinfo server and handle the boot scenarios above.

   If bootinfo tells the loader to boot from a disk without any additional
   command line arguments, then pxeboot loads the first sector of that
   partition (or MBR) and jumps to it. Job done. If there is a command line,
   and we are booting FreeBSD, then the loader can directly boot the
   kernel (first arg), after first parsing the remaining arguments
   converting key=val strings into loader variables that can be passed to
   the kernel. This is how delay nodes get booted--we specify either a
   custom kernel name or we pass "kern.hz=10000" as an argument.
   However, if it is a Linux kernel being booted, then the presense of a
   command line will likely cause the boot to fail. You cannot even specify
   an alternate kernel. (Something to keep in mind if we ever want a Linux
   delay node!) Note that the only case where a Linux command line works
   is if the on-disk boot loader is LILO. pxeboot has code to pass LILO
   command line arguments. But only very ancient images have LILO boot blocks.

   Otherwise pxeboot uses TFTP to load a kernel and "mfsroot". pxeboot will
   only load FreeBSD-based MFSes because it only knows how to direct boot a
   FreeBSD kernel. Thus the FreeBSD pxeboot must be paired with FreeBSD MFSes.
   MFS booting basically works by fooling pxeboot into thinking that, e.g.,
   boss:/tftpboot/frisbee is the root filesystem. Hence it tries to read the
   normal boot time things out of "/boot", including loader.conf and the
   kernel. loader.conf contains special variables to tell it to use a file as
   an in-memory root filesystem ("mfsroot"). So pxeboot.emu reads all these
   files from "the filesystem" which is handled behind the scenes via the
   libstand TFTP code. Once the kernel is booted, the OS looks pretty normal,
   modulo the fact that the command set is extremely limited and the disk is
   extremely small.

2. The "Linux MFS" BIOS boot path.

   Back around 2008, Ryan Jackson made a valiant effort to move us out of
   the FreeBSD world and into Linux by creating a boot chain that used
   Linux-related tools. These include a version of Grub (pre-2.0) modified
   to support our bootinfo protocol, a custom grub.cfg to handle booting
   from a partition or an MFS, a busybox-based Linux filesystem that serves
   as the MFS, and a Linux 3.2.7 kernel.

   PXE downloads the custom Grub (/tftpboot/pxeboot_grub2pxe) specified
   in the DHCP config file. It uses /tftpboot/grub2/grub.cfg as its initial
   config. This is the custom script which invokes bootinfo and then either
   boots from a partition or loads an MFS.

   For the MFS case, it reloads a config file from one of
   {admin,frisbee,newnode}_linux and that config file loads the Linux kernel
   and initrd (MFS). The kernel is passed a special "elab_mode=" command
   line parameter set to one of "frisbee", "admin", or "newnode" so that
   only one kernel and MFS is needed for all three uses. This one MFS is a
   much more complete system than what any of the FreeBSD MFSes provide.
   However, it is busybox so many of the standard commands are subsets of
   the real versions.

   For booting a FreeBSD system from disk, grub uses the kFreeBSD command
   to load /boot/loader from the disk passing along environment variables
   via kFreeBSD.* variables.

   *** TODO: figure out exactly what command lines we can handle and
       finish this section. ***

   For booting from disk without a command line in Linux, we typically just
   chainload the partition boot block. ...

   Of note for chain booting. In order to pass (command line?) arguments to
   the FreeBSD bootloader from Grub, we had to hack a special version of the
   BSD bootloader for some older FreeBSD images. This is /boot/emuboot in those
   images. I am not sure if this is needed for newer versions of the laoder.

3. The Moonshot m400 ARM U-boot/pxelinux boot path.

   In 2014, we got the HP Moonshot ARM cartridges that use U-boot and
   PXE boot via a builtin version of pxelinux. Since there is no working
   version of FreeBSD for these boxes, we use Linux-based MFSes--one an
   initramfs (frisbee) and the other an NFS-mounted root filesystem (admin).

   When booted, the nodes still DHCP, but they now ignore the "filename"
   argument that is returned. Instead, the first contact is via a TFTP read
   of a file from /tftpboot/pxelinux.cfg. There is a sequence of files it
   attempts to read starting with the very specific (a file name that is
   the same as the MAC address) up to a generic "default". We use the
   individual files named after the MAC that are cloned from a template
   file /tftpboot/pxelinux.cfg/boot.template. This template has different
   menu entries for booting from the disk (via the boot partition which is
   also the root partition for us), booting from NFS (the admin MFS), or
   booting from an initramfs (frisbee MFS). It also has a special PXEWAIT
   entry, but we won't talk about that. Whenever a node changes its boot
   designation (e.g., set/clear node_admin, during reloading) we create
   a new version of the template with the correct default boot menu item
   (and correct NFS root FS path for NFS booting).

   One interesting aspect of the PXE boot is that, since bootinfo is not
   called on the boot path, we handle sending the initial PXEBOOTING and
   BOOTING state transition events via dhcpd. The dhcpd.conf file has an
   "on commit" section that allows us to invoke a script whenever a client
   has accepted a lease. In our case we call /usr/testbed/sbin/reportboot
   which will send the appropriate events. Since the Utah Moonshot cluster
   now supports both ARM and x86 (see #5 below), we have to be careful to
   only send these events from dhcpd for U-boot nodes.

   The frisbee MFS is a more conventional Linux initramfs, but it is
   utterly unrelated to the "Linux MFS" (#2) that we use on the x86 nodes.
   Trying to recreate Ryan's build environment for the ARM architecture was
   a non starter. Instead, I started with the default "initrd" and just
   kept adding stuff til rc.frisbee worked! So it is just as, or even more,
   limited than the FreeBSD frisbee MFS.

   I took a different tack for the admin MFS. Rather than trying to build
   up the initramfs further (with perl, etc.) I took advantage of two things:
   1) the fact that we had an NFS bootable version of Ubuntu 14 that we got
   from HP and 2) the fact that we are using ZFS on the fs node making cloned
   filesystems fast and easy. The z/nfsroot/m400 zfs, mounted at /nfsroot/m400,
   is the base volume for the m400 admin FS. z/nfsroot/m400@current is the
   snapshot which is cloned on demand to create filesystems for the individual
   nodes; e.g., z/nfsroot/ms0102 would be the volume for ms0102 when it is
   in admin mode. So the node_admin command will "zfs clone" a new version
   of the snapshot when admin mode is entered, and "zfs destroy" when done.
   This is triggered if the osid for the admin MFS contains the path

4. The Intel NUC got-a-lame-BIOS boot path.

   In 2015, we got some Intel NUCs for PhantomNet. The plan here was just
   to use the traditional boot path (#1) but we ran into a problem where,
   when configured for legacy BIOS and booting via the network, the SATA
   (AHCI?) option ROM (aka, driver) was not loaded. So we could not access
   the disk from the PXE booted loader. This is fine for the MFSes, which
   never access the disk until the OS is loaded, but made it impossible to
   boot from a disk partition since we could not load the boot sector.

   Fortunately, Grub2 has the option to use "native" device drivers rather
   than the BIOS, so using the Linux MFS (#2) seemed feasible...except that
   the pre-2.0 version of Grub Ryan used did not support native SATA. Thus
   I embarked on a project to figure out how Ryan built pxeboot_grub2pxe
   and transfer his changes into Grub 2.02. The result is now in the repo. To not clash
   with the older grub-pre-2 install, we set the filename differently in
   dhcpd.conf. For these nodes we use /tftpboot/grub2pxe-native-vga/grub2pxe
   as the filename. That directory also contains a further tweaked version
   of grub.cfg which is configured to use native disk drivers via a variable
   setting at the top of the file. The naming convention for directories:
   grub2pxe--, allows us to have a different grub.cfg in
   each, specifying the drivers to use ("bios" or "native") and the console
   ("vga", "sio1", "sio2") via variables. They share modules, fonts, etc.
   via symlinks to the grub2.0 directory.

   The MFSes used are just the VGA versions of the ones discussed in #2
   (/tftpboot/admin_linux_vga and /tftpboot/frisbee_linux_vga -- I hope
   you were not looking for a consistent naming convention from me!)

   Booting from disk however is harder. We cannot just chainload the on-disk
   loader since it most likely expects to use the BIOS disk interface.
   So for Linux, we just boot the kernel directly. We load the grub.cfg
   file from the disk and hope the config file is compatible with our
   version of Grub! For FreeBSD...well, we just don't boot FreeBSD on
   these nodes. Grub2 is capable of directly booting FreeBSD kernels,
   so it would be possible.

5. The Moonshot m510 x86 UEFI-only boot path.

   Now we are in 2016 and we have yet another variant! The new HP Moonshot
   x86 cartridges are UEFI only, no legacy BIOS setting. So it is time to
   suck it up and figure out how to handle UEFI. The goal is to have common
   images that boot on both BIOS and UEFI machines. We can actually pull this
   off if we never want to boot directly from the disk. When booting from
   disk with UEFI, you not only have to have a boot loader that speaks the
   UEFI API but also a dedicated boot partition formatted with a FAT
   filesystem that contains the boot loader. Making that work, along with
   having an MBR boot block with a BIOS-speaking boot loader was going to
   be a task. Fortunately, it is simpler if we assume that we always boot
   from the network (we do). Now we just have to load an EFI-savvy boot
   loader via PXE (which UEFI still supports) and then (hopefully) just
   use that to directly load/boot the kernel and MFS. My travails are
   documented in

   We wound up needing to use Grub here as well for the first level boot.
   But of course Grub has to be compiled differently to get EFI support.
   So now we have a new "driver" catagory: "efi". For these particular
   nodes the DHCP "filename" is set to /tftpboot/grub2pxe-efi-sio1/grub2pxe.
   And grub.cfg needed some further tweaks. To boot FreeBSD from disk,
   we just need to chainload /boot/loader.efi from disk. It exists in
   all our existing images for newer versions of FreeBSD.

   Since I just could not stomach the prospect of upgrading the Linux MFS,
   or at least the kernel, to support newer hardware (NVME disk, Mellanox
   10Gb Ethernet), I took advantage of Grub's support for booting FreeBSD
   and figured out how to load a FreeBSD kernel and MFS with Grub.

How to make things.

1. pxeboot.emu, the FreeBSD-based PXE boot loader.

   pxeboot.emu is still built using the FreeBSD 7.2 sources from along with  
   a tweaked version of libstand. IMPORTANT: these changes only exist in
   that source tree (and the corresponding RCS directories). They have
   never been isolated and put in a git repo anywhere! To rebuild pxeboot,
   you will have to allocate a node running the FBSD72-STD image (a 32-bit
   version of FreeBSD 7.2--anybody see any weak links in this chain?),
   login and:

   mount -o ro fs:/share/freebsd/7.2/src /usr/src
   cd /usr/src/lib/libstand
   make obj all install
   cd /usr/src/sys/boot
   make obj all
   cd /usr/obj/usr/src/sys/boot/i386/emuboot
   cp pxeboot 

   By default this will create the "sio1" serial port console version.
   To create "null", "vga", "sio[234]" versions you will need to tweak
   /share/freebsd/7.2/src/sys/boot/i386/ as follows:

   "vga" version: comment out:

   BTX_SERIAL=                1

   and build.

   "null" version: uncomment:

   #CFLAGS+=               -DNONINTERACTIVE

   To be safe, also comment out the serial console lines as per the "vga"
   build above.

   "sio2" verison: make sure the three serial console lines above are
   *not* commented out and then uncomment the "2f8" line from:


   "sio3" version: include serial console lines, uncomment the "3e8" line.

   "sio3" version: include serial console lines, uncomment the "2e8" line.

   That is it! You should now have all five versions of pxeboot.

2. The FreeBSD MFSes.

   The FreeBSD MFSes were hand-rolled long ago (circa 2000) and have been
   lovingly maintained ever since. They all started life as FreeBSD 6(?)
   filesystems. Since then, they have been hand updated a couple of times.
   Once to move them to FreeBSD 8 binaries and once to create a version
   with 64-bit binaries. They are waaay past due for another update, or
   even better, a reproducible reimplementation using NanoBSD or something.

   The disk-loading ("frisbee") MFS was mercilessly (and non-systematically)
   hacked to remove all packages and standard binaries that were not needed
   to run our rc.frisbee script. It is truly a unicorn--a 12MB filesystem
   (4MB compressed) that is fast to download with TFTP. In fact, it is now
   smaller than the kernel that is downloaded with it. The frisbee MFS does
   NOT include perl or ssh among many other utilities normally thought of as
   The admin ("freebsd") MFS was basically the same, but without taking out
   the more useful things like perl and ssh. It is a 40MB filesystem.

   The "newnode" MFS, used only when adding new nodes to the testbed, is very
   similar to the admin MFS--basically the same with different Emulab apps

   "Rebuilding" these basically means updating the Emulab client-side that
   is installed. Start with a FBSD83-STD or FBSD83-64-STD image and copy over
   the "mfsroot" files from /tftpboot/{frisbee,freebsd,freebsd.newnode}/boot.
   Then set up an Emulab clientside build tree ala:

   mkdir obj; cd obj
   /clientside/configure --with-TBDEFS=/defs-utahclient

   and for each MFS (e.g. frisbee):

   mdconfig -a -t vnode -f mfsroot.frisbee -u 5
   mount /dev/md5 /mnt
   setenv DESTDIR /mnt
   make frisbee-mfs-install
   umount /dev/md5
   mdconfig -d -u 5

   And then copy back the updated mfsroot file. There are make targets
   "mfs-install" (admin MFS) "newnode-mfs-install" (newnode MFS) for the

   If you only need to update a shell/perl script, then you don't need
   a FreeBSD 8 node. You can just mount up the MFS and copy in the script
   by hand.

   It is possible to pair one of a FreeBSD 8, 9, or 10 kernel with the
   mfsroot. We have had to use increasingly newer kernels to pick up
   support for newer hardware over time.

   These kernels used to be of the custom unicorn variety, reflecting just
   the hardware we had, but nowadays we use a variant of the GENERIC config
   so we get a more broad range of hardware coverage. There should be a
   {i386,amd64}/conf/TESTBED-PXEBOOT-GENERIC configuration file to use.
   This includes a custom Emulab enhancement (ipod) as well as additional
   drivers and setting we need. To build the kernel, start with a node
   running the appropriate image and then:

   mount fs:/share/freebsd//src /usr/src
   cd /usr/src
   cd /usr/obj/usr/src/sys/TESTBED-PXEBOOT-GENERIC
   cp kernel 

3. The Linux MFS.

   Swap in Mike's emulab-ops/Linux-MFS experiment.

   Do magic shit.
   (I think there is a README out in the persistent blockstore that has the

4. The HP Moonshot m400 MFSes.

   How to update the initramfs.
     - unwrap from u-boot, unpack, update, re-pack, rewrap for u-boot
     - all in

   How to update the NFS filesystem.
     - install new stuff into /nfsroot/m400 on fs
       (no makefile target yet, I don't think)
     - move old @current snapshot to @old
     - create new @current snapshot

   How to update the kernel.
     - don't! I tried once and broke it.
     - need to have NFS (NFSROOT?) built into the kernel

5. Building "grub2pxe", the modified version of Grub for PXE boot.

   Hopefully you will never have to build the 1.97+ version Ryan used
   for the  original Linux MFS environment. The source repo is:
   and I have it checked out in ops:~mike/grub-ryan.

   For the 2.02 version, the repo is
   and there is a file that describes how to build and install it.