Kb55

Emulab FAQ: Software Setup: What if I need more disk space on my nodes?

Each node has a partition at the end of the disk that you can use if you wish. Regardless of the OS in use, it will always be DOS partition 4. However, the naming convention differs between OSes and even between versions of the OS. In Linux the partition is likely to be either /dev/hda4 (PATA) or /dev/sda4 (SATA or SCSI), though it is possible that it will be on /dev/sdb instead. in FreeBSD, it is probably one of /dev/ad0s4 (PATA), /dev/ad4s4 (SATA), or /dev/da0s4 (SCSI or SAS). One slightly more generic way to determine the name is to do "df /" to see where the root filesystem is. The extra partition will be on the same disk, but as partition 4 rather than 1 or 2. There is no filesystem on this extra partition, so you'll need to create it yourself as described below.

Before going any further, there is one very important point: anything you put in this disk space will be lost when your experiment swaps out! That is, unless you create a Custom disk image before it gets swapped. You can also use the loghole utility to retrieve information from your nodes before swap-out. Use:

loghole sync -r remotedir

to specify the directory you create on the experiment nodes, as described below. Loghole uses rsync over ssh to collect files from your experiment nodes, so you can run it multiple times as your experiment progresses and it will do minimal work to update the results. Running as a post-process or during idle times avoids mucking up your experiment timings with NFS overhead, if that matters. It puts results under /proj/pid/exp/eid/logs (where pid and eid and the project and experiment IDs) in per-node subdirectories, and has some support for archiving previous states as zip files.

Making a filesystem.

The standard FreeBSD and Linux images also include a script, /usr/testbed/bin/mkextrafs to make the remaining space on the root disk (partition 4) available to you. Just do the following as root:

/usr/testbed/bin/mkextrafs /mnt

and it will create the filesystem, mount it on /mnt and make an entry in /etc/fstab so that the filesystem will be mounted on future reboots.

mkextrafs can also be used to create a filesystem on the second disk (for machines that have one, see the table below):

/usr/testbed/bin/mkextrafs -s 0 -r da1 /disk2

If that script does not exist, you can perform the steps by hand. For example, in Linux do the following as root:

• Set the correct partition type (type 83 - ext2fs): fdisk /dev/hda
  • Press 't' to change partition sysid.
  • Enter '4' as the partition to change.
  • Enter '83' to specify ext2fs.
  • Type 'w' to save and exit.
• Create the filesystem: mkfs /dev/hda4
• Mount the filesystem: mount /dev/hda4 /mnt You may want to add the filesystem to /etc/fstab so that it will be automatically mounted on future reboots.

In FreeBSD, do the following as root:

• First, you need to set the correct partition type: fdisk -i4 /dev/ad0
  • Do not change what the BIOS thinks.
  • Edit the partition info, setting the sysid to 165; leave other metrics alone.
  • Do not change the active partition
  • Write out the new partition table
• Next, you have to create a BSD disklabel on the partition: disklabel -w ad0s4 auto
• Create the filesystem: newfs /dev/ad0s4c
• Finally, mount it: mount /dev/ad0s4c /some/where You may want to add the filesystem to /etc/fstab so that it will be automatically mounted on future reboots.

Finding available space on nodes.

The available space in the extra partition ranges from 0-200+GB depending on the disk type and the OS loaded. Some of the node types also have a second hard drive that can be used (after following the procedure described above to create a filesystem on /dev/sdb (Linux) or /dev/ad5 or /dev/da1 (FreeBSD). In the current Utah Emulab:

  Node Type    Disk Type    4th partition     Extra disk  Extra disk name

  pc600        PATA         0-6 GB            None
  pc850        PATA         27-33 GB          None
  pc2000       PATA         7-13 GB           20 GB       ad5 (BSD), sdb (Linux)
  pc3000       SCSI         133-139 GB        146 GB	  da1, sdb
  d710         SAS          237-243 GB        250 GB	  da1, sdb
  d820	       SATA/SAS	    237-243 GB	      6 x 600 GB  mfid[1-6], sd[b-g]