Instructions for updating your boss/ops nodes from FreeBSD 7.3 to 8.3.
The overview is:
* update core FreeBSD to 8.3
* update your ports
* update the Emulab software
NOTE: if you have an install older than FreeBSD 7, then do NOT attempt
a direct upgrade to FreeBSD 8. You may need to first incrementally upgrade
your OS to FreeBSD 7 and then follow this procedure. We (testbed-ops @ Utah)
do not have much experience with this--google is your friend.
NOTE: this assumes a combined ops and fs node. If you have seperate ops
and fs machines, contact testbed-ops@flux.utah.edu.
NOTE: if you are upgrading the OS, you MUST upgrade the Emulab software
to at least the emulab-stable repository as of 12/23/12! At the very least,
there are fixes to Emulab scripts so that they will work with the newer
ports that are being installed (e.g., perl 5.12).
Updating FreeBSD will be done with freebsd-update. Be sure to read:
http://www5.us.freebsd.org/doc/handbook/updating-upgrading-freebsdupdate.html
before starting to understand what is going to happen.
-1. BACKUP!
Make sure you have backed up critical non-Emulab stuff. Ideally, you
have some persistent backup for both your boss and ops, but if not then
here are some important directories to save that will have files modified
in one way or another:
/etc
/usr/local/etc
/var/named
/var/db
Note however that without a full backup, once you install the 8.x
system binaries (step 6b), there is no going back! The above directories
just contain files (e.g., /var/named/named.conf) that may not get merged
correctly and that you might have to go back and look at to fix it.
0. Shutdown the testbed on boss:
sudo /usr/testbed/sbin/testbed-control shutdown
Start with boss first, but the same steps apply to ops as well unless
otherwise noted.
1. Since I am assuming csh syntax throughout, first do this:
cd /root
sudo tcsh
# if you are an emacs user
setenv EDITOR emacs
2. You will need a temporary directory with at least 2GB of disk space.
Identify that location and:
setenv TMPDIR /q/tmp
cd $TMPDIR
3. Make a copy of your /etc directory in case things get badly confused.
Most of the files that will need to be merged between your current system
and FreeBSD 8 are in this directory. Also, make a list of the current
installed packages.
cp -rp /etc $TMPDIR/etc
pkg_info > $TMPDIR/ports.old
4. Make sure you have a GENERIC kernel installed as described in the FreeBSD
document above. If you don't have one, grab the one from Emulab:
fetch -o $TMPDIR http://www.emulab.net/downloads/GENERIC.7.3.tar.gz
tar -xzf $TMPDIR/GENERIC.7.3.tar.gz -C /boot
If your /boot/kernel is actually a GENERIC kernel, then copy it:
cp -rp /boot/kernel /boot/GENERIC
5. Run freebsd-update to update everything. freebsd-update wants to use
/var/db/freebsd-update as its state directory, and requires at least 1GB
of space to do a full update. If you don't have 1GB in your /var filesystem
you will need to modify /etc/freebsd-update.conf and set WorkDir to a
subdir in your $TMPDIR and create that subdir:
sed -i .bak -e "/^#.*WorkDir/s;.*;WorkDir $TMPDIR/freebsd-update;" \
/etc/freebsd-update.conf
mkdir -p $TMPDIR/freebsd-update
It isn't clear if this state should really be persistent, so I don't know
the consequences of removing the WorkDir later. Anyway, now run:
freebsd-update -r 8.3-RELEASE upgrade
Note that this will take quite a while and also the upgrade will not
clobber any installed files.
You will likely be prompted to resolve some merge conflicts between your
current versions and FreeBSD 8 versions of files. For the most part,
the "correct path" should be obvious. Here are some hints:
/etc/{passwd,master.passwd,group}: there will be a new ID "hast" in the
new version; keep this AND accounts in your version.
/etc/hosts, /etc/mail/*: mostly just CVS ID string diffs; for these
you should accept the new versions to simplify future upgrades.
/etc/ssh/*: keep the Emulab settings for PermitRootLogin (yes) and
Protocol (2,1).
/etc/syslog.conf: keep the Emulab version.
/var/db/namedb/named.conf: (on boss) This one you will need to merge
by hand; I cannot give you any general advise here other than
you need to keep the two "views".
NOTE WELL: if it asks you if something is reasonable, don't say "no"
or freebsd-update will exit unfinished. You will just have to remember
what it screwed up and fix it up later:
/etc/ntp.conf: it will replace the Emulab content with the new content.
You will need to go back and put back the Emulab lines (leave the
header comment block from the new one).
Once it reaches the point where it gives you a list of files it will
update, etc. and you are in "more", it is safe to quit (q) out of that
as it is the last thing (unless you want to see it!)
6. Run freebsd-update to actually install things. This is done in two
phases.
6a. First the GENERIC kernel is updated after which you must reboot
into that kernel:
freebsd-update install
cd /boot
mv kernel kernel.bak
mv GENERIC kernel
shutdown -r now
# ...wait for reboot and then login...
sudo tcsh
# reset tmpdir to whatever you chose above
setenv TMPDIR /q/tmp
# for emacs-ians
setenv EDITOR emacs
*** THIS IS A CRITICAL POINT. ***
Once you run the next freebsd-update, there is no going back! It will
install an 8.x userland and you will no longer be able to boot the 7.x
kernel in any meaningful way.
At this point you should make sure that all the filesystems were found and
correctly mounted. This is because the behaviour of the GEOM subsystem
changed between 7.x and 8.x w.r.t. an MBR with BSD disklabeled partitions.
Disks with GPT partitions are safe. One thing you can do is to fsck all
filesystems in read-only mode:
fsck -n
This can take a long time and can report some inconsistencies (missing
blocks, bad summary info, unrefed files) since the filesystems are active.
However it should NOT report any read errors. If it does, then you have
a problem.
The problem with MBR-based partition is as follows. It has always been the
case that a BSD disklabel cannot have a start + size, measured in sectors,
larger than 4 billion (i.e., they are 32-bit unsigned values).
However, previously the start of a label was relative to the MBR partition
it was contained in, so a BSD partition could be up to four billion sectors
(2TB) if it started at the beginning of the enclosing MBR partition (i.e.,
at start==0), even if that MBR partition started one billion sectors into
the disk. Now the start of a partition is an absolute sector number, so that
same 2TB BSD partition would now have a start value of one billion instead
of zero and would be flagged as invalid and be inaccessible.
6b. If all is well, you run freebsd-update again to install the userland:
freebsd-update install
# there is no going back, so get rid of the old kernel
rm -rf /boot/kernel.bak
7. At this point you need to update the ports.
7a. The first step is to deal with the old ports tree, either moving it
out of the way or just removing it. We do it in the background since
it will take awhile and won't otherwise affect us:
# get it out of the way quickly
mv /usr/ports /usr/ports.bak
# if you care about the old ports tree (unlikely)
mv /usr/ports.bak $TMPDIR/ports &
# if you don't (likely)
rm -rf /usr/ports.bak &
7b. And now download the "known good" ports tree:
fetch -o $TMPDIR http://www.emulab.net/downloads/FreeBSD-8.3-ports.tar.gz
# this will take a while
tar -xzf $TMPDIR/FreeBSD-8.3-ports.tar.gz -C /usr
7c. And download the pre-built packages:
cd $TMPDIR
fetch http://www.emulab.net/downloads/FreeBSD-8.3-packages.tar.gz
tar -xzf FreeBSD-8.3-packages.tar.gz
setenv PKG_PATH $TMPDIR/packages
8. Reinstall the ports. With an FreeBSD 8.3 boss/ops, we have switched to
perl 5.12 (from 5.10), python 2.7 (from 2.6), apache 2.2 (from 1.3),
and php 5.3 (from 5.2).
I explored trying to upgrade the set of ports with portupgrade,
but it quickly became a nightmare of individual updates and selective
de-installation. So the approach taken here is to just remove all existing
ports and install the new set.
8a. First get rid of some cruft in your make.conf:
sed -i '.bak' \
-e '/PYTHON_DEFAULT_VERSION/d' \
-e '/PERL_VERSION/d' \
-e '/OVERRIDE_LINUX/d' \
-e '/ENABLE_SUIDPERL/d' \
-e '/^# added by/d' /etc/make.conf
8b. Out with the old...
Make sure you are already root as "sudo" will get deinstalled in this
step. Then do:
pkg_delete -a
It is likely to complain about a lot of stuff: could not completely remove
directories, checksums that don't match, non-existent directories, etc.
While you might be left with some cruft in your filesystem, everything is
okay as long as doing:
pkg_info
shows that there are no packages installed.
You may be tempted at this point to just remove /usr/local/* but DO NOT.
There are still many things left in /usr/local that are valuable, in
particular any config info for packages that you are going to reinstall
(e.g., apache).
8c. ...in with the new...
For ops:
# the base package
pkg_add emulab-ops-5.0
# the fs stuff
pkg_add emulab-fs-5.0
# add assorted useful things like vim and emacs
pkg_add emulab-extras-1.0
For boss:
# the base package
pkg_add emulab-boss-5.0
# if you are a protogeni site
pkg_add emulab-protogeni-3.0
# add assorted useful things like vim and emacs
pkg_add emulab-extras-1.0
Once again, there will be a lot of chatter from pkg_add. As long as
the command exits with a status of zero (do "echo $?" immediately after
the command) you are okay.
8d. Re-install other local ports. You will have to compare the list you
made earlier in $TMPDIR/ports.old to what is currently installed and see
what is missing. This sounds easy, but really isn't--a simple
before-and-after diff won't work since the versions of most all installed
packages will have changed and just about every line will be different.
The set of dependent packages may have changed as well, so packages that
were previously installed may just no longer be necessary. Here is a first
cut apporach to diffing the old and new ports:
sed -E -e 's;^([^ ]+).*;\1;' -e 's;(.*)-[^-].*;\1;' ports.old > OLD
env -u PKG_PATH pkg_info > $TMPDIR/ports.new
sed -E -e 's;^([^ ]+).*;\1;' -e 's;(.*)-[^-].*;\1;' ports.new > NEW
diff OLD NEW
This attempts to strip off the version number for ports to make a diff
more meaningful. Here are a couple of packages that you do NOT need for
sure:
linuxthreads # use the native threads instead
mrouted # frisbee has its own IGMP querier now
py26-* # should be replaced by py27-* equivs
xz # now part of the base system
However, at the end of the day, you will still have to use your judgement
here...sorry!
9. Download, build and install the new Emulab software.
9a. Download the git repo.
setenv REPO emulab-stable # or emulab-devel
setenv ESRC /usr/testbed/src/testbed # or whereever your source tree is
setenv EOBJ /usr/testbed/obj/testbed # or whereever your build tree is
mv $ESRC $ESRC.bak
git clone http://git-public.flux.utah.edu/git/$REPO.git $ESRC
9b. Configure the new version. You will need the defs-* file you
used when you configured Emulab the last time. If this was in your old
Emulab source tree, you will want to copy that into the new tree.
IMPORTANT NOTE: you need to add TBUSEDBI=1 to your defs file! Otherwise
things will not run.
setenv DEFS <full-path-to-your-defs-file>
echo TBUSEDBI=1 >> $DEFS
mv $EOBJ $EOBJ.bak
mkdir $EOBJ
cd $EOBJ
$ESRC/configure --with-TBDEFS=$DEFS
9c. Apply a critical perl patch before attempting the install:
# boss and ops - fix to perl Self Loader
patch -d /usr/local/lib/perl5/5.12.4 -i $ESRC/patches/SelfLoader.patch
9d. Build and install the Emulab software:
For ops:
gmake opsfs-install
For boss there is a bit of a dance, as the install wants to first apply
updates but some of the updates rely on new versions of scripts (with
TBUSEDBI set) already being installed. So we first force an install
without updates and then apply the updates:
gmake
gmake boss-install-force
gmake update-testbed-nostop
10. Apply a couple of fixups.
# ops, if you installed mailman - patch mailman
patch -f -l -p0 -i $ESRC/patches/mailman.patch
# ops - new enable vars for samba and quotas
sed -i .bak -e \
'/smbd_enable="YES"/s//samba_enable="YES"; nmbd_enable="NO"/' /etc/rc.conf
sed -i '' -e '/enable_quotas/s//quota_enable/' /etc/rc.conf
# boss and ops - unpack crypto egg
set os=`uname -r`
set egg="M2Crypto-0.21.1-py2.7-freebsd-$os-i386.egg"
mv /usr/local/lib/python2.7/site-packages/$egg /var/tmp
easy_install -N -H None -Z /var/tmp/$egg
mv /var/tmp/$egg /usr/local/lib/python2.7/site-packages/$egg.bak
unset os egg
*** UTAH ONLY ***
There are .utah versions of the boss/ops config files that need
to be installed down in the apache/v2 directory. The GNUmakefile
doesn't do this.
*** END UTAH ONLY ***
# boss - fixup apache
cd $EOBJ/apache
gmake install
cp -rp /usr/local/etc/apache/ssl* /usr/local/etc/apache22
rm -f /usr/local/etc/rc.d/{apache.sh,.apache.real}
sed -i .bak -e 's/apache_/apache22_/' /etc/rc.conf
# ops - fixup apache
cd $EOBJ/apache
gmake control-install
cp -rp /usr/local/etc/apache/ssl* /usr/local/etc/apache22
rm -f /usr/local/etc/rc.d/{apache.sh,.apache.real}
sed -i '' -e 's/apache_/apache22_/' /etc/rc.conf
ln -sf data-dist /usr/local/www/data
11. And a final call to freebsd-update to clear out old shared libraries
that ports may have been using:
freebsd-update install
If you have older 7.x binaries that you need to run (i.e., your user's
binaries), you will need to install the compat7.x port. Probably you
will only want this on your ops node:
pkg_add compat7x-i386-7.3.703000.201008_1
12. If you are using a custom kernel, now would be the time to rebuild
and install it. Note that you may have to update your config in
/usr/src/sys/i386/conf to account for new drivers, features, etc.
Compare your config with the current version of whatever config you
derived from to get a sense of these differences.
<update config file in /usr/src/sys/i386/conf>
cd /usr/src
make -j8 buildkernel KERNCONF=<config name>
mv /boot/kernel /boot/GENERIC
make installkernel KERNCONF=<config name>
13. When you are all done with both ops and boss, reboot them both.
14. If things come up okay, login to boss and turn the testbed back on:
sudo /usr/testbed/sbin/testbed-control boot
