Skip to content. | Skip to navigation

Personal tools


You are here: Home / Wiki / Appendix / upgrade-7x-83.html


Instructions for updating your boss/ops nodes from FreeBSD 7.3 to 8.3.
The overview is:

 * update core FreeBSD to 8.3
 * update your ports
 * update the Emulab software

NOTE: if you have an install older than FreeBSD 7, then do NOT attempt
a direct upgrade to FreeBSD 8. You may need to first incrementally upgrade
your OS to FreeBSD 7 and then follow this procedure. We (testbed-ops @ Utah)
do not have much experience with this--google is your friend.

NOTE: this assumes a combined ops and fs node. If you have seperate ops
and fs machines, contact

NOTE: if you are upgrading the OS, you MUST upgrade the Emulab software
to at least the emulab-stable repository as of 12/23/12! At the very least,
there are fixes to Emulab scripts so that they will work with the newer
ports that are being installed (e.g., perl 5.12).

Updating FreeBSD will be done with freebsd-update. Be sure to read:
before starting to understand what is going to happen.


   Make sure you have backed up critical non-Emulab stuff. Ideally, you
   have some persistent backup for both your boss and ops, but if not then
   here are some important directories to save that will have files modified
   in one way or another:


   Note however that without a full backup, once you install the 8.x
   system binaries (step 6b), there is no going back! The above directories
   just contain files (e.g., /var/named/named.conf) that may not get merged
   correctly and that you might have to go back and look at to fix it.

0. Shutdown the testbed on boss:

   sudo /usr/testbed/sbin/testbed-control shutdown

Start with boss first, but the same steps apply to ops as well unless
otherwise noted.

1. Since I am assuming csh syntax throughout, first do this:

   cd /root
   sudo tcsh

   # if you are an emacs user
   setenv EDITOR emacs

2. You will need a temporary directory with at least 2GB of disk space.
   Identify that location and:

   setenv TMPDIR /q/tmp
   cd $TMPDIR

3. Make a copy of your /etc directory in case things get badly confused.
   Most of the files that will need to be merged between your current system
   and FreeBSD 8 are in this directory. Also, make a list of the current
   installed packages.

   cp -rp /etc $TMPDIR/etc
   pkg_info > $TMPDIR/ports.old

4. Make sure you have a GENERIC kernel installed as described in the FreeBSD
   document above. If you don't have one, grab the one from Emulab:

   fetch -o $TMPDIR
   tar -xzf $TMPDIR/GENERIC.7.3.tar.gz -C /boot

   If your /boot/kernel is actually a GENERIC kernel, then copy it:

   cp -rp /boot/kernel /boot/GENERIC

5. Run freebsd-update to update everything. freebsd-update wants to use
   /var/db/freebsd-update as its state directory, and requires at least 1GB
   of space to do a full update. If you don't have 1GB in your /var filesystem
   you will need to modify /etc/freebsd-update.conf and set WorkDir to a
   subdir in your $TMPDIR and create that subdir:

   sed -i .bak -e "/^#.*WorkDir/s;.*;WorkDir $TMPDIR/freebsd-update;" \
   mkdir -p $TMPDIR/freebsd-update

   It isn't clear if this state should really be persistent, so I don't know
   the consequences of removing the WorkDir later. Anyway, now run:

   freebsd-update -r 8.3-RELEASE upgrade

   Note that this will take quite a while and also the upgrade will not
   clobber any installed files.

   You will likely be prompted to resolve some merge conflicts between your
   current versions and FreeBSD 8 versions of files. For the most part,
   the "correct path" should be obvious. Here are some hints:

   /etc/{passwd,master.passwd,group}: there will be a new ID "hast" in the
       new version; keep this AND accounts in your version.

   /etc/hosts, /etc/mail/*: mostly just CVS ID string diffs; for these
       you should accept the new versions to simplify future upgrades.

   /etc/ssh/*: keep the Emulab settings for PermitRootLogin (yes) and
       Protocol (2,1).
   /etc/syslog.conf: keep the Emulab version.
   /var/db/namedb/named.conf: (on boss) This one you will need to merge
       by hand; I cannot give you any general advise here other than
       you need to keep the two "views".

   NOTE WELL: if it asks you if something is reasonable, don't say "no"
   or freebsd-update will exit unfinished. You will just have to remember
   what it screwed up and fix it up later:

   /etc/ntp.conf: it will replace the Emulab content with the new content.
       You will need to go back and put back the Emulab lines (leave the
       header comment block from the new one).
   Once it reaches the point where it gives you a list of files it will
   update, etc. and you are in "more", it is safe to quit (q) out of that
   as it is the last thing (unless you want to see it!)

6. Run freebsd-update to actually install things. This is done in two

6a. First the GENERIC kernel is updated after which you must reboot
    into that kernel:

   freebsd-update install
   cd /boot
   mv kernel kernel.bak
   mv GENERIC kernel
   shutdown -r now
   # ...wait for reboot and then login...

   sudo tcsh
   # reset tmpdir to whatever you chose above
   setenv TMPDIR /q/tmp
   # for emacs-ians
   setenv EDITOR emacs


   Once you run the next freebsd-update, there is no going back! It will
   install an 8.x userland and you will no longer be able to boot the 7.x
   kernel in any meaningful way.

   At this point you should make sure that all the filesystems were found and
   correctly mounted. This is because the behaviour of the GEOM subsystem
   changed between 7.x and 8.x w.r.t. an MBR with BSD disklabeled partitions.
   Disks with GPT partitions are safe. One thing you can do is to fsck all
   filesystems in read-only mode:

   fsck -n

   This can take a long time and can report some inconsistencies (missing
   blocks, bad summary info, unrefed files) since the filesystems are active.
   However it should NOT report any read errors. If it does, then you have
   a problem.

   The problem with MBR-based partition is as follows. It has always been the
   case that a BSD disklabel cannot have a start + size, measured in sectors,
   larger than 4 billion (i.e., they are 32-bit unsigned values).
   However, previously the start of a label was relative to the MBR partition
   it was contained in, so a BSD partition could be up to four billion sectors
   (2TB) if it started at the beginning of the enclosing MBR partition (i.e.,
   at start==0), even if that MBR partition started one billion sectors into
   the disk. Now the start of a partition is an absolute sector number, so that
   same 2TB BSD partition would now have a start value of one billion instead
   of zero and would be flagged as invalid and be inaccessible.

6b. If all is well, you run freebsd-update again to install the userland:

   freebsd-update install

   # there is no going back, so get rid of the old kernel
   rm -rf /boot/kernel.bak

7. At this point you need to update the ports.

7a. The first step is to deal with the old ports tree, either moving it
   out of the way or just removing it. We do it in the background since
   it will take awhile and won't otherwise affect us:

   # get it out of the way quickly
   mv /usr/ports /usr/ports.bak

   # if you care about the old ports tree (unlikely)
   mv /usr/ports.bak $TMPDIR/ports &

   # if you don't (likely)
   rm -rf /usr/ports.bak &

7b. And now download the "known good" ports tree:

   fetch -o $TMPDIR
   # this will take a while
   tar -xzf $TMPDIR/FreeBSD-8.3-ports.tar.gz -C /usr

7c. And download the pre-built packages:

   cd $TMPDIR
   tar -xzf FreeBSD-8.3-packages.tar.gz
   setenv PKG_PATH $TMPDIR/packages

8. Reinstall the ports. With an FreeBSD 8.3 boss/ops, we have switched to
   perl 5.12 (from 5.10), python 2.7 (from 2.6), apache 2.2 (from 1.3),
   and php 5.3 (from 5.2).

   I explored trying to upgrade the set of ports with portupgrade,
   but it quickly became a nightmare of individual updates and selective
   de-installation. So the approach taken here is to just remove all existing
   ports and install the new set.

8a. First get rid of some cruft in your make.conf:

   sed -i '.bak' \
     -e '/PERL_VERSION/d' \
     -e '/OVERRIDE_LINUX/d' \
     -e '/ENABLE_SUIDPERL/d' \
     -e '/^# added by/d' /etc/make.conf

8b. Out with the old...

   Make sure you are already root as "sudo" will get deinstalled in this
   step. Then do:

   pkg_delete -a

   It is likely to complain about a lot of stuff: could not completely remove
   directories, checksums that don't match, non-existent directories, etc.
   While you might be left with some cruft in your filesystem, everything is
   okay as long as doing:


   shows that there are no packages installed.

   You may be tempted at this point to just remove /usr/local/* but DO NOT.
   There are still many things left in /usr/local that are valuable, in
   particular any config info for packages that you are going to reinstall
   (e.g., apache).

8c. with the new...

   For ops:

   # the base package
   pkg_add emulab-ops-5.0
   # the fs stuff
   pkg_add emulab-fs-5.0

   # add assorted useful things like vim and emacs
   pkg_add emulab-extras-1.0

   For boss:

   # the base package
   pkg_add emulab-boss-5.0

   # if you are a protogeni site
   pkg_add emulab-protogeni-3.0

   # add assorted useful things like vim and emacs
   pkg_add emulab-extras-1.0

   Once again, there will be a lot of chatter from pkg_add. As long as
   the command exits with a status of zero (do "echo $?" immediately after
   the command) you are okay.

8d. Re-install other local ports. You will have to compare the list you
   made earlier in $TMPDIR/ports.old to what is currently installed and see
   what is missing. This sounds easy, but really isn't--a simple
   before-and-after diff won't work since the versions of most all installed
   packages will have changed and just about every line will be different.
   The set of dependent packages may have changed as well, so packages that
   were previously installed may just no longer be necessary. Here is a first
   cut apporach to diffing the old and new ports:

   sed -E -e 's;^([^ ]+).*;\1;' -e 's;(.*)-[^-].*;\1;' ports.old > OLD
   env -u PKG_PATH pkg_info > $TMPDIR/
   sed -E -e 's;^([^ ]+).*;\1;' -e 's;(.*)-[^-].*;\1;' > NEW
   diff OLD NEW

   This attempts to strip off the version number for ports to make a diff
   more meaningful. Here are a couple of packages that you do NOT need for

       linuxthreads	# use the native threads instead
       mrouted		# frisbee has its own IGMP querier now
       py26-*		# should be replaced by py27-* equivs
       xz		# now part of the base system

   However, at the end of the day, you will still have to use your judgement

9. Download, build and install the new Emulab software.

9a. Download the git repo.

   setenv REPO emulab-stable		# or emulab-devel
   setenv ESRC /usr/testbed/src/testbed # or whereever your source tree is
   setenv EOBJ /usr/testbed/obj/testbed # or whereever your build tree is
   mv $ESRC $ESRC.bak
   git clone$REPO.git $ESRC

9b. Configure the new version. You will need the defs-* file you
   used when you configured Emulab the last time. If this was in your old
   Emulab source tree, you will want to copy that into the new tree.
   IMPORTANT NOTE: you need to add TBUSEDBI=1 to your defs file! Otherwise
   things will not run.

   setenv DEFS <full-path-to-your-defs-file>
   echo TBUSEDBI=1 >> $DEFS

   mv $EOBJ $EOBJ.bak
   mkdir $EOBJ
   cd $EOBJ
   $ESRC/configure --with-TBDEFS=$DEFS

9c. Apply a critical perl patch before attempting the install:

   # boss and ops - fix to perl Self Loader
   patch -d /usr/local/lib/perl5/5.12.4 -i $ESRC/patches/SelfLoader.patch

9d. Build and install the Emulab software:

   For ops:

   gmake opsfs-install

   For boss there is a bit of a dance, as the install wants to first apply
   updates but some of the updates rely on new versions of scripts (with
   TBUSEDBI set) already being installed. So we first force an install
   without updates and then apply the updates:

   gmake boss-install-force
   gmake update-testbed-nostop

10. Apply a couple of fixups.

   # ops, if you installed mailman - patch mailman
   patch -f -l -p0 -i $ESRC/patches/mailman.patch 

   # ops - new enable vars for samba and quotas
   sed -i .bak -e \
    '/smbd_enable="YES"/s//samba_enable="YES"; nmbd_enable="NO"/' /etc/rc.conf
   sed -i '' -e '/enable_quotas/s//quota_enable/' /etc/rc.conf

   # boss and ops - unpack crypto egg
   set os=`uname -r`
   set egg="M2Crypto-0.21.1-py2.7-freebsd-$os-i386.egg"
   mv /usr/local/lib/python2.7/site-packages/$egg /var/tmp
   easy_install -N -H None -Z /var/tmp/$egg
   mv /var/tmp/$egg /usr/local/lib/python2.7/site-packages/$egg.bak
   unset os egg
   *** UTAH ONLY ***
   There are .utah versions of the boss/ops config files that need
   to be installed down in the apache/v2 directory. The GNUmakefile
   doesn't do this.
   *** END UTAH ONLY ***

   # boss - fixup apache
   cd $EOBJ/apache
   gmake install
   cp -rp /usr/local/etc/apache/ssl* /usr/local/etc/apache22
   rm -f /usr/local/etc/rc.d/{,.apache.real}
   sed -i .bak -e 's/apache_/apache22_/' /etc/rc.conf

   # ops - fixup apache
   cd $EOBJ/apache
   gmake control-install
   cp -rp /usr/local/etc/apache/ssl* /usr/local/etc/apache22
   rm -f /usr/local/etc/rc.d/{,.apache.real}
   sed -i '' -e 's/apache_/apache22_/' /etc/rc.conf
   ln -sf data-dist /usr/local/www/data

11. And a final call to freebsd-update to clear out old shared libraries
   that ports may have been using:

   freebsd-update install

   If you have older 7.x binaries that you need to run (i.e., your user's
   binaries), you will need to install the compat7.x port. Probably you
   will only want this on your ops node:

   pkg_add compat7x-i386-7.3.703000.201008_1

12. If you are using a custom kernel, now would be the time to rebuild
   and install it. Note that you may have to update your config in
   /usr/src/sys/i386/conf to account for new drivers, features, etc.
   Compare your config with the current version of whatever config you
   derived from to get a sense of these differences.

   <update config file in /usr/src/sys/i386/conf>
   cd /usr/src
   make -j8 buildkernel KERNCONF=<config name>
   mv /boot/kernel /boot/GENERIC
   make installkernel KERNCONF=<config name>

13. When you are all done with both ops and boss, reboot them both.

14. If things come up okay, login to boss and turn the testbed back on:

   sudo /usr/testbed/sbin/testbed-control boot