Making boss/ops/fs/etc. packages:
[ Updated 11/05/2013 for 9.2 and 9.2 64-bit packages. ]
0a. Make sure the ports tree in /share/freebsd/X.X/ports has the current
Emulab ports. They are all in the "misc" subdir. If they are out of
date, update them and remake the ports.tar.gz tarball.
0b. Allocate a node on which to build everything and load the appropriate
OS on it. Pick a d710 if possible, both for the speed and the second disk.
I try to use pc599 if available because it has more processors and RAM than
the other d710s.
1. Create a couple of extra filesystems for src/ports and the build tree.
I put these on different disks for better performance:
<login to the node>
sudo mkextrafs /repos
sudo mkextrafs -r /dev/da1 -s 0 /usr/obj
sudo rmdir /usr/src /usr/ports
sudo ln -s ../repos/src /usr/src
sudo ln -s ../repos/ports /usr/ports
sudo mkdir /repos/src /repos/ports /repos/tmp
sudo chmod 1777 /repos/tmp
# make sure soft updates are enabled
sudo umount /repos
tunefs -n enable /dev/da0s4e
sudo mount /repos
sudo umount /usr/obj
tunefs -n enable /dev/da1s1e
sudo mount /usr/obj
2. Install the src and ports trees. From the node do:
scp fs:/share/freebsd/9.2/src.tar.gz /repos/tmp/
scp fs:/share/freebsd/9.2/ports.tar.gz /repos/tmp/
sudo tar xzf tmp/src.tar.gz
sudo tar xzf tmp/ports.tar.gz
Note that we avoid using NFS do to the race condition. We scp instead.
Note also that the FreeBSD source tree (/usr/src) is really only needed
to build the "lsof" package in emulab-extras.
3. Deinstall all existing ports. You will want to make sure you are root first,
since this will get rid of "sudo".
4. Be paranoid, clear out old state.
Look in /var/db/pkg. The only thing in that directory after doing the
"pkg_delete -a" should be the pkgdb.db file (and maybe pkgdb.fixme?)
If there is anything else, get rid of it.
Remove everything in /var/db/ports.
I have never been bold enough to remove everything in /usr/local, but
in the latest build I did remove some stuff:
rm -rf python* compat/pkg/* perl5* ruby*
rm -rf cups
rm -rf share/* man/* info/* libdata/ldconfig32
If you are building ports on 8.3 and above, edit /etc/make.conf and add:
# for 8.3, or...
# ...for 9.2
5. Now begin the tedious, iterative process. First, do:
setenv PACKAGES /usr/obj/ports/packages
setenv TMPDIR /repos/tmp
The former ensures that packages wind up in a single directory. The latter
will force the compiler (and other utilities) to use /repos/tmp for
Now for each emulab-* port you install, you will need to perform the ritual
steps below. But before doing these for any port, first check for any
port-specific notes below (5a-5h). The ritual:
rm -f /usr/obj/ports/packages/All/*
# This step appears to not be strictly necessary.
# If you are really paranoid (and have lots of time) you can do this.
From ops do:
scp -p pcXXX:/usr/obj/ports/packages/All/\* .
Note that the deinstall doesn't undo the effect of the install, it just
deinstalls (really, unregisters) the meta-port so that the package-recursive
will work. The "make clean" avoids complaints about work directories
5a. Do emulab-boss. Notes:
* Boost. We use the boost-all package which includes docs. I generally
prefer not to drag around the docs, so I go into devel/boost-all, do
a "make config" and deselect the docs box. Now when you do the make
in emulab-boss, it should use that config when it gets to boost.
* Git. Go into the devel/git directory, do a "make config" and deselect
the GUI box. Otherwise it will drag in possibly conflicting versions of
TCL and TK.
* Pubsub. If the pubsub libraries are installed but were installed by hand
and not through the port, then they will cause the port to not be
installed as part of the emulab-boss port (since the dependency in the
boss Makefile will be satisfied). This will cause an error when creating
packages. The easiest fix is to just remove /usr/local/lib/libpubsub*,
which is what the Makefile checks for.
Additional 9.2 issues:
* Curl. Defaults to building with GSSAPI which does not exist in the
base system. Should it? I don't know, but I just go into ftp/curl
and do "make config" to remove that option.
* When doing "make install", graphviz wanted pango which wanted cairo.
The pango build blew up because it claims to need cairo with glib
enabled. But the default config does have glib enabled and retrying
the make worked fine. I think this is a case of stale state where
something that was not true when the pango build started was true
later on. I fixed the Makefiles for this and another port with a
related problem, see /share/freebsd/9.2/README.ports for details.
5b. Do emulab-ops.
* Assuming you built emulab-boss already, everything should be fine.
Otherwise you will need to deal with git and pubsub as above.
5c. Do emulab-fs.
5d. Do emulab-protogeni.
5e. Do emulab-tipserv.
5f. Do emulab-mailman.
5g. Do emulab-extras. Notes:
* We do this one last so that portaudit doesn't get installed til the
end. Otherwise, it would cause builds of vulnerable ports to fail.
See Step 6 below if installing vulnerable ports bothers you.
* Emacs. I used to build the -nox11 varient so we didn't drag in a ton
of crap, but Leigh prefers the real deal and came up with a reasonable
set of options. These are now baked into the emulab-extras Makefile.
If you get an error from the Makefile about "m17n support requires Xft"
then you probably have an old options file. You should remove
* Lsof. Remember to install the FreeBSD source in /usr/src as talked about
back on step #1. Otherwise, the build of lsof is going to fail.
* Vim. Yes, you are not imagining that, it really is downloading 400+
patches. Swearing will make you feel better and pass the time...
* If you wind up with the ruby+nopthreads port, then again you probably
have some left over /var/db/ports/*/option files. This isn't a show
stopper, so probably not worth fixing after the fact.
5h. Painful post-port processing.
* I don't always do it, but after all the ports are installed
run portaudit and see what security vulnerabilities there are. Manually
update the ports and make notes in /share/freebsd/x.x/README.ports.
This is fraught with peril however. Some ports versions are assumed in
various other ports and things can degenerate rapidly when you try to
do this! See one of the existing README.ports files for examples.
* Install the bsdadmintools port (sysutils/bsdadmintools) and run
"pkg_libchk" to ensure that all the library dependencies of the ports
are correctly resolved.
6. After you have collected all the ports, you should have seven directories
full of packages on ops: boss, ops, fs, protogeni, tipserv, mailman, extras;
which we will now aggregate into one set.
Most of the subdirs will include packages like perl which should in theory
be the same, and for certain will be if you don't do "pkg_delete -a" between
making each one. However, just in case there are differences, I use the
strategy I have always used, which is to favor the version built as part of
the "more important" package. So to create the aggregate package I do
foreach d (extras mailman tipserv protogeni fs ops boss)
mv $d/* .
7. Now you should have a fully functional set of packages. The last thing to
do is make sure they install. ON YOUR BUILD NODE (NOT on boss or ops!) do
the following and watch for any errors (there will be plenty of
# make sure you are not on boss or ops!
hostname -s | grep -Eq '^(boss|ops)$' && \
(echo 'STOP! you are on boss or ops, ^C now!'; sleep 100000; exit)
# all possible boss-install packages
# all possible ops-install packages
pkg_add cvsd* # may already be installed
pkg_add adodb* # may already be installed
# all possible fs-install packages
# all (someday) possible tipserv-install packages