Updating the Testbed
This document describes the general procedure for upgrading your existing Emulab. The instructions apply only if:
- You are tracking the current Emulab source code via GIT, and
- You are currently running from emulab-devel later then May 15th, 2010, or you from emulab-stable later them July 13th, 2010 (tag stable-20100714), and
- You have already upgraded to FreeBSD 6.3 on your boss/ops/fs servers.
If you are running a previous version of Emulab, and you plan to update to the current version of emulab-devel or emulab-stable, we advise you to do this in two stages:
- First update to emulab-stable tag "stable-20100525" using the older updating instructions.
- Then update to the most current release of emulab-stable or emulab-devel using the instructions on this page.
The second stage is actually quite easy, so if you make it through the first stage, the rest is a breeze.
First a note on upgrading your Emulab. On the whole, the Emulab software that runs on boss and ops, can be upgraded independently of the disk images that run on your experimental nodes. Swapins and swapouts get locked out while the server update is happening, but if things go well, it takes only a few minutes. When updating the disk images, running experiments are not affected; the new disk images are just there waiting for the next experiment swapin, or for a swapped in experiment to explicitly ask to be reloaded.
There are dozens of sites running the Emulab software, and there are a diversity of ways that they choose to handle it. In Utah, we run on 'the bleeding edge' of our development repository, meaning that we sometimes update the server side of the software multiple times per day. A few other sites who want to get new features all the time, update from emulab-devel every few days or so. Most choose to track emulab-stable, and do not necessarily update every time we push new code to emulab-stable (about every 2-3 months), unless there is something in the new code they really want to have.
In general, the more often you update, the easier it tends to be.
Okay, on with show ...
Unpack the Emulab software. You should do this in a directory accessible to your boss and ops machines (and your fs and tipserv machines if you have them) to make the various installs easier. Typically this would be a directory in /users or /proj.
If you doing this install via a GIT checkout, then do your checkout to a location in /users or /proj.
Look at doc/UPDATING in the source tree to find out what you have to update in the boss/ops FreeBSD environment before updating the Emulab software. Entries are timestamped, so look for things that have changed since the last time you installed. Some entries are marked as needing to be done after install. Skip those but remember them for later.
BACKUP YOUR DATABASE on boss. Do:
boss> mysqldump tbdb > db.backup
If you need to upgrade the Emulab Server Packages for this release, follow the directions in Step 1 of Installing Emulab Software on Boss (add the '-f' option to the pkg_add command to force an upgrade).
Configure and build an object tree. This should be done in a different directory then your source tree:
boss> cd /your/objdir boss> /your/srcdir/configure --with-TBDEFS=<your-defs-file> <options> boss> gmake clean boss> gmake
This step is optional; if you are interested in what the upgrade is going to do (besides updating the SQL schema), then you can do a dry-run. If you want to just plow ahead, then skip to the next step.
Note that a wide terminal screen, say about 120 characters, will make it easier to view the results. To do a dry run:
boss> cd /your/objdir/install boss> perl update-testbed -n
The output will give you a somewhat detailed idea of the changes that will be made.
The next step is to run the update. The current procedure allows for Boss to be updated without requiring that it be rebooted. The update script will first disable web logins, wait for your testbed to settle down (if experiments are in the process of swapping), halt all testbed daemons, and then install the new software. At the end of the process, the testbed daemons will be restarted and the web interface disabled. In some cases, you will be told reboot Boss, but that is generally a rare case.
To run the update: Note that you probably want to use the script command to make a record of the output, in case there is a problem and you need to send a transcript to Utah.
boss> script /tmp/myscriptfile Script started, output file is /tmp/myscriptfile boss> cd /your/objdir boss> sudo gmake update-testbed boss> exit Script done, output file is /tmp/myscriptfile
Typical output looks like:
Updating the testbed ... ** Putting the testbed to sleep before updating ... -> Turning off the web interface and disallowing swaps. -> Waiting a few seconds for testbed to quiet down ... -> Looking for experiments still in transition. ** Testbed is quiet; stopping testbed daemons ... ** Testbed is stopped. Proceeding to update Running DB updates ... Running pre-install updates ... Installing testbed software ... this will take a while! Output saved to /var/tmp/update.73082 Running post-install updates ... Starting up testbed daemons dbboot bootinfo tmcd capd lastlogd sdcollectd stated sslxmlrpc_server reloadd checkupd poold mysqld_watchdog expire_daemon sa_daemon checknodes_daemon batchd wrapper Turning on the web interface and allowing swaps
If you have one or more "tipserv" machines, login to them, cd to your build directory and do:
tipserv> sudo gmake tipserv-install
If you need to upgrade the Emulab Server Packages for this release, follow the directions in Step 2 of Installing Emulab Software on Ops (add the '-f' option to the pkg_add command to force an upgrade). This only needs to be done on the ops server.
If you have a combined ops/fs machine (the common case), skip to step 10. Otherwise:
- Login to "fs", cd to your build directory and do:
fs> sudo gmake fs-install
- Login to "ops" ("users"), cd to your build directory and do:
ops> sudo gmake ops-install ops> cd rc.d; sudo gmake control-install
Skip to Step 11.
If you need to upgrade the Emulab Server Packages for this release, follow the directions in Step 2 of Installing Emulab Software on Ops (add the '-f' option to the pkg_add command to force an upgrade).
Login to your "ops" ("users") machine, cd to your build directory and do:
ops> sudo gmake opsfs-install
Please look at the current copyright on your site to make sure it is acceptable.
This rest of this step is optional and necessary only if you have local content on your site that wish to protect with your own copyright statement. If you are satisfied with the copyright contained in copyright-standard.html, then you can skip to the next step.
To install your own copyright statement:
boss> cd /your/srcdir/www boss> cp copyright-local.txt copyright-local.html
Now edit copyright-local.html and fill in values for @SITENAME@, @SITEDATES@, @SITECOPYRIGHT@. Next install the new file:
boss> cd /your/objdir/www boss> sudo gmake install
If you were told to reboot Boss above, then also reboot the tipservers and ops.