The "pc3000" Nodes

Machines and Interfaces

The "pc3000" machines are Dell PowerEdge 2850s with a single 3GHz processor, 2GB of RAM, and 2 10,000 RPM 146GB SCSI disks. Each has 4 available Gigabit experimental net interfaces, however due to lack of Gb switch ports, not all are usable at Gb speeds. Instead, they are a mix of 100Mb and 1Gb.

Also, due to the addition of NetFPGA cards and the need to fit a 10Gb interconnect into the switch chassis, not all nodes have all four interfaces connected as they originally did. The distirbution of Gb ports now looks like:

Note that all 160 machines have at least 1 Gb interface and all can be used for 100Mb as well.

There is also a 5th Gb interface on each machine, that is used to pairwise connect adjacent machines. For example, pc201 is connected directly to pc202 via a cross over cable. These interfaces are not yet available, pending further support in the resource mapper.

All machines have serial console lines accessible from users.emulab.net via the "console" command or remotely via the tiptunnel interface.

Images and Kernel Support

Currently, the Emulab standard FreeBSD (FBSD410-STD, FBSD64-STD, FBSD72-STD, FBSD80-STD), Linux (RHL90-STD, FC4-STD, FC8-STD, FC10-STD, "legacy" RHL73-STD), and Windows (WINXP-UPDATE-pc3000, WINXP-SP{0,1,2}-pc3000) images will run on the new machines.

There are also a small set of standard images with 64-bit versions of the OSes that will run on these machines: FBSD72-64-STD, FEDORA8-64-STD.

If you have built custom images based on our standard images before Sept 2 2005, they will likely not work on the new machines due to a lack of the correct disk driver. You will either need to re-customize based on the current images or modify your existing image to add the correct SCSI driver. For BSD, add the following to your kernel config file:

    device          mpt

For Linux, add the following to your .config:

    CONFIG_FUSION=y
    CONFIG_FUSION_BOOT=y
    CONFIG_FUSION_MAX_SGE=40
    CONFIG_FUSION_ISENSE=m

NOTE: The default Pro/1000 ethernet driver (e1000) included in the RedHat 9.0 kernel sources does not work on the pc3000s. This includes the emulab-patched RHL90 source tree! When creating a custom RH9-era kernel, you must build and install an updated version of the e1000 driver module. You can get it here: /share/linux/e1000-5.5.4.tar.gz . Please read the README inside this tarball for build/install instructions.

For Windows you will need to re-customize based on our current -pc3000 images.

Caveats

• No bandwidth shaping between 100Mb and 1000Mb. We have not yet looked at providing bandwidth shaping values between 100Mb and 1000Mb.
• The "direct connect" 5th interface is not usable. As mentioned, we need modifications to the resource mapper and there are also security issues to be resolved.
• Interswitch bandwidth may limit topologies in non-intuitive ways. Since the Gb interfaces and the 100Mb interfaces are on different switches, and the new machines are on different switches than the old pc600 and pc850 machines, the bandwidth of the interswitch links can have a significant impact on whether an experimental topology can map or not.
• No vnode characterization has been done. The so-called "colocation factor" has been arbitrarily set to 75 right now. It will likely be clamped down considerably.
• There might be assorted failures due to scaling issues. Experiments of over 150 machines, which were hard to do before, may reveal further scaling issues that will need to be addressed. Expect an increase in unexpected swapin failures as a result.
• These machines are radically different than the old machines. Since these machines are so much faster, have so much more and faster memory, have so much more and faster disk, results of your experiments on these machines could be dramatically different. That is, if your applications were CPU, memory, or disk intensive. If you are interested in reproducing old results, you might want to limit your experiments to using only pc600 and pc850 nodes.
• Hyperthreading is available, but SMP support is not currently compiled in to the stock Emulab images (with the exception of RHL73-STD). If you want it, you will have to compile your own kernel with SMP enabled and create a disk image. Note that FreeBSD 4.x SMP kernels will panic on a uniprocessor machine, so a combined 4.x image that runs on all Emulab nodes is not currently possible. We plan to upgrade our Linux and FreeBSD 5 images to support SMP in the near future as well as address the 4.x SMP issue for standard images.
• The network interfaces use "interrupt throttling." The Intel Pro1000 cards used for the experimental interfaces in these machines default to using interrupt throttling. This will add latency to packet delivery and reception. See the knowledge base entry for how to change or disable this feature.