Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Wiki / Windowsimagecreation

Windowsimagecreation

Creating A Windows OS Image for Your Emulab Site

Creating A Windows OS Image for Your Emulab Site

Note: This is not a document on the use of Windows on Emulab. It is for admins looking to create a Windows image.
For the former, go here

This page serves to describe the process, caveats, tricks, and black magic needed to bring up a Windows OS image at your Emulab site. We've tried to make the process as turnkey as possible.  However, some manual steps are required.

Knowledge Prerequisites

 
 This document is thorough, but it is not a script:  You have to do a bit of thinking and interpolating now and then.

Emulab Know-how

 

Please make sure you are familiar with Emulab's disk imaging system and have a good grasp of Emulab administration in general.    Here are some documentation pointers:

Toto, I've a feeling we aren't in Unix anymore!

Welcome to the wide world of Windows.  Make sure you have a solid understanding of Windows OS concepts before proceeding: the registry (and the different registry hives), device (driver) management, WAIK/Sysprep/Windows Setup, Event logs, WMI, CIFS, Windows services, and licensing mechanisms. You should also have reasonable familiarity with the command line ('net', 'ipconfig', 'netsh', 'reg', 'sc', 'tasklist', 'wmic', etc.). Knowledge of group policy and active directory isn't necessary (for now).

Windows Knowledge Links:

Since much of Emulab's client-side runs on top of Cygwin under Windows, a good understanding of Cygwin is also important. More info can be found on the Cygwin website:

Licensing and Software Acquisition

Windows is commercially licensed software. Please make sure that you have the necessary licenses/rights to install and use it, and that you are abiding by all applicable licensing agreements. The Flux group cannot distribute Windows or any other Microsoft product.

Supported Versions

Windows

The Emulab client-side is only known to work with Windows versions specifically mentioned in this section. Use other versions at your own risk/peril.

  • Windows 7 Professional Edition 32-bit (service pack 1) - English language.

Emulab Server-Side

Any Emulab version released from 2012 on should be compatible on the server-side.  Earlier versions of Emulab on the server-side have bugs related to handling very large image files, and Windows images are very large indeed.  Your MFS images need to date from the 2012 era onward as well.

Emulab Client-Side

Any Emulab source released/dated after mid-November 2012 should work for the client-side (Windows node). Any 'emulab-stable' release used MUST be dated after mid-November 2012. Any git checkouts of 'emulab-devel' MUST include all updates through mid-November 2012 at a minimum.

Tools Required

The following resources will be needed for a successful build of Windows for Emulab:

  • A Windows "admin workstation"
    • Somewhere you can install and use tools (e.g. WSIM), stage and extract files, etc.
    • This machine does not have to be part of your testbed.
  • A bootable Windows OS DVD disc (or ISO image if you will be using remote media).
  • A USB Memory stick, formatted FAT32.
  • A node in your Emulab testbed with hardware that is natively supported by Windows.
    • I.e., the processor, chipset, control network NIC, disk controller, etc. are supported "in-box" by the version you are installing.
    • BIOS must be able to boot DVDs (or remote DVD ISO images).
    • Must have access to the keyboard, video, and mouse! A remote KVM with virtual media capabilities is very handy here.
    • The ability to reach the Utah Emulab site over the Internet is strongly recommended (http).
  • License keys and running license server.
    • E.g. a KMS server or Multiple Activation Key plus VAMT proxy service.
  • The Windows Driver Kit - use version 7.10
  • The Windows Automated Installation Kit (WAIK).
    • Please use WAIK for Windows 7 / Server 2008.
    • Needed for the Windows System Image Manager (WSIM) tool. The latest version should be fine.
  • An Emulab source tree (dated mid-November 2012 or later).
  • Your customized Emulab definitions file
  • The OpenSSH host keys used on the nodes at your site.

The Setup Process

If you are performing a from-scratch build of a Windows image, go consecutively through the phases. For updating your image, skip to phase 5.

Phase 0: Enable Windows on the Server Side

Your server-side components need to be setup to support Windows. Perform the following steps:

  • Check for Windows support. On your ops node, run:
      ps ax | grep smbd
    If you don't see any Samba smbd processes, you most likely don't have Windows support enabled.
  • Enable Windows support, if necessary:
    • Locate your Emulab defs file, and add "WINSUPPORT=1" on its own line.
    • Reconfigure, recompile and re-install your Emulab server software using the updated definitions file
      • For this, use the source tree that matches your current installation! An accidental upgrade on the server side is probably not what you want...
      • More info: Updating Your Testbed
    • Check again, as above to see if 'smbd' processes are running on your ops server.

If things are working correctly, your ops node should be offering up shares in the "EMULAB" Windows Workgroup. Note that most shares are hidden (non-browsable). Remember that your Windows password is different from your regular Emulab password. It can be managed along with the rest of your profile via the web interface.

Phase 1: Setup License Management

You are free to use whatever method you wish to manage your license keys and activation. Note that Emulab does not manage license keys or services for you. Please follow all applicable license agreements.

Phase 2: Configure and Prep

The idea here is to add site-specific Windows information to your Emulab definitions file and then further customize an unattended answer file. You will need to define:

  • WINSUPPORT=1

Simply needed to enable Windows components in the source tree

  • WINDOWS_ORGNAME

Set this to the name of your organization/institution.

  • WINDOWS_OWNER

Optional. This will default to WINDOWS_ORGNAME if not set in your definitions file.

  • WINDOWS_TIMEZONE

Set this to your local timezone. A list of Windows timezones can be found here.

  • WINDOWS_KEY_<VERSION>="<a license key string>"

where <VERSION> corresponds to the version of Windows you are using (see table). You will most likely set this to a MAK that you own, or KMS setup key.

Windows Version Autoconf Variable
Windows 7 Professional 32-bit WINDOWS_KEY_7PRO_X86

After setting these configuration directives, configure your source tree with your updated definitions file. The source tree you use here needs to be dated mid-November 2012 or later!

On your ops node, do:

mkdir winclient-obj
cd winclient-obj
<path_to_source>/clientside/configure --with-TBDEFS=<path_to_your_windows_defs_file>

You should now have a semi-customized unattended setup file located here: 'winclient-obj/tmcc/cygwin<BASE_VERSION>/unattend-<VERSION>'. See the table directly below for details.

Windows Version object directory unattended setup file name
Windows 7 Pro 32-bit tmcc/cygwinseven unattend-7pro-x86.xml

Copy the indicated unattended setup file to your Windows Admin Workstation. Mount your Windows installation media (DVD or ISO image). Start WSIM, and point it at the "install.wim" file in the "sources" subdirectory of your installation media, and then pick your target version of Windows in the subsequent dialog box. Now open up the unattended setup (answer) file you just copied over. You should see something like this:

WSIM-Emulab.png

Look through the answer file and familiarize yourself with the settings. Modify the 'root' user's password, located here:

Components -> oobeSystem -> Microsoft-Windows-Shell-Setup_neutral -> UserAccounts -> LocalAccounts -> LocalAccount[Name="root"] -> Password

Make certain to remember this password. You will need it later.

There are a great many unattended answer file settings that you can add or tweak within WSIM. Tread carefully, however, as this answer file has been crafted to work with these instructions and generally makes the resulting installation Emulab friendly/compatible.

  • If you are using a KMS server, you will want to specify the KMS configuration details now in this answer file.
  • Look into the settings available for adding out-of-box (third-party) drivers. Initially, you should be using hardware that is supported natively by your target version of Windows. Once you have an image up and running on this hardware, come back and look into adding drivers (as needed) for other hardware types in your testbed that are not supported in-box. Check with your hardware vendor to verify support and get drivers.

Save the changes to your unattended answer file, then copy it to the root of the FAT32-formatted USB memory stick and rename it to: Autounattend.xml. Windows setup will look for this file in the root directory of any connected removable storage device.

  • (Optional) Copy your Emulab source to the USB memory device.

You may find it convenient to have your Emulab source tree on the USB memory stick as well. This would be a good time to copy it there. It should be the same source you used to create the unattended setup file.

Phase 3: Bring Up a Fresh Windows Install

The goal of this phase is to install Windows on bare metal. You should have already identified a suitable target node (see the Tools Required section above).

  • Setup a single node experiment.

Specifying the node you identified as a compatible destination for the version of Windows you have selected, create a single node experiment. Set the OS to and existing one that boots from the first partition (e.g. FBSD*-STD). As admin, set the experiment to be unswappable, idle ignore, no max duration, etc. You don't want anyone or anything yanking this node out from under you while doing this work!

  • Connect hardware to the target node and mount installation media.

If you have a remotely accessible KVM, connect it to the target node now. Mount the installation media, either as a virtual drive via the remote KVM, or as a physical DVD. Plug in the USB memory stick you prepared earlier.

  • Boot from the installation media

Grab the VGA console, (re)boot the node, and force it to boot from the attached Windows installation media. A Windows PE environment should fire up and automatically read the Autounattend.xml file located on the removable USB memory device you plugged in. Windows setup may ask you to pick which version of Windows to install - select the same one you've been using throughout this process. After several minutes of churning and a couple of reboots, the node should finally come up to the login "welcome" screen.

  • Login as the 'root' user and run Windows Update

Use the root password you set in the unattended answer file in Phase 2. Assuming you don't want security holes and such, run Windows Update to patch the OS current.

  • (Optional) Clean up after Windows Update

To reduce the size of your image, have Windows commit service packs and remove backup files. From an elevated command prompt, run:

dism /online /cleanup-image /spsuperseded
  • Cleanly shutdown your Windows node

Shutdown the node in anticipation of creating an image of the base installation. From a command prompt: 'shutdown -t 0', or use the GUI.

  • Create an image of the base OS installation.
From your Emulab installation's web GUI, browse to the "Create a new Image Descriptor (EZ Form) page": Experimentation (drop down) -> List ImageIDs -> Create an Image Descriptor

Sample settings:

baseimage-descriptor.png

Things to note:

  • Project: This should be set to the project you are building the node under.
  • Descriptor Name: Set to something relevant. Putting the hardware type in the name is a good idea. (Note: 'D710' is a Utah-specific hardware type.)
  • Node Types: Set this to the target node's type. Uncheck all other types.
  • Node to obtain snapshot from: Leave this blank, as shown.

After customizing the fields appropriately, click 'submit' to create the descriptor.  Click "OK" when warned that you have not specified a node to obtain a snapshot from.  Once the image descriptor has been successfully created, browse back to it and click the "Snapshot Node Disk into Image" menu option.  Specify the node you've been using to build the image, and click "go".  After imaging completes, the node should boot back up into Windows.

N.B.: This image will be bound to the hardware type of the target node. Trying to boot it on a different node type will likely fail.

N.B.: Since Windows needs more space, a larger partition has to be used (50GB) compared to the size of a 'normal' Emulab partition (6GB). This requires a different MBR, and hence the 'full disk image' setting even though only partition 1 is relevant here.

Phase 4: Prepare Windows for the Emulab Client-side

This phase largely consists of running some canned scripts provided in the Emulab source tree.

  • Login as the root user and bring up an "admin" command shell (from the VGA console).

Supply the password you set earlier while modifying the unattended setup answer file. Once logged in, launch an "admin" command shell by clicking the 'start' button, typing in 'cmd', right clicking on the "cmd" program entry in the search results, and selecting "run as Administrator".

  • Set powershell to allow running unsigned scripts.

From the "admin" command shell, do:

powershell -Command "Set-ExecutionPolicy Unrestricted"
  • Gain access to your Emulab source tree on the target node.

Via CIFS, the USB memory stick, etc. This should be the same client-side source tree you've been using above. You can copy it to some location on the node itself if you wish.

  • Run the first set of actions to configure the Windows OS

From a command shell, do:

cd <path_to_your_emulab_source>\clientside\tmcc\cygwin<BASE_VERSION>
powershell .\runsequence.ps1 -actionfile basecmds-phase1.af

Where <BASE_VERSION> is as in the table above for the path to OS-specific files in the Emulab source for the version of Windows you are installing. The script will ask for a password for the 'root' account. Type in what you used above while configuring your unattended answer file. You will see output on the console indicating what the command sequence is doing. There is also a log file located here: C:\Windows\Temp\runsequence.log. Ignore any warnings about the root account already existing.

  • Reboot to apply settings

When the machine comes back up, User Access Controls will be disabled, so all future commands will effectively run with "Administrator" rights.

  • Run the second set of actions to install and perform initial configuration of Cygwin:

From a command shell, do:

cd <path_to_your_emulab_src>\clientside\tmcc\cygwin<BASE_VERSION>
powershell .\runsequence.ps1 -actionfile basecmds-phase2.af

You will be asked to supply a password for the Cygwin SSH account. Use whatever you like, and jot it down somewhere secure for future reference. The installation of Cygwin will take quite a while (As a point of reference, it takes about 25 minutes on a D710 node at Utah with local package sources).

N.B.: If your target node can't directly access the Utah Emulab site (mirror.emulab.net and www.emulab.net) via http, you will need to copy the entire Cygwin package repository kept there to somewhere that the node can access. Adjust the paths in the above 'basecmds-phase2.af' command sequence file as well. Do not try to use an arbitrary Cygwin mirror!

  • Install your site's OpenSSH keys

Start a Cygwin shell and shutdown ssh with:

cygrunsrv -E sshd

Next, copy your site's OpenSSH host keys (ssh_host*key*) to /etc (C:\Cygwin\etc). Restart OpenSSH:

cygrunsrv -S sshd

Check to make sure ssh started back up (look at the tasklist or 'ps -Welf' output). Don't know where to find your SSH keys? Just copy them from another swapped-in node running your stock Linux or FreeBSD image.

  • Extract the "devcon.exe" tool from the Windows Driver Kit and place in C:\Windows\System32

On your admin workstation you should already have the Windows Driver Kit downloaded. Create a disc and insert it in your AW, or mount the ISO. Next, install the WDK and then search for devcon.exe. Alternatively, just extract devcon as follows from the source:

msiexec.exe /a "<path_to_WDK_files>\setuptools_x86fre.msi" targetdir="C:\Devcon"

Note that the version of the WDK you use should match the version of Windows you are installing (x86 or x64).

  • (Optional) Install other software

If there is other Windows-compatible software that you feel is of general interest/use to your users, this is a good time to install it. The Utah Emulab site installs: 7zip, NTemacs, and Wireshark. The Cygwin setup executable is in C:\Software\Cygwin. Run it with the '-X' switch (disables signature checking on the setup.ini file), accept the defaults for package sources, and then pick through and add whatever else you wish. Other Cygwin packages the Utah Emulab site installs: subversion, ncurses (to get the 'clear' command).

  • (Optional) Create a checkpoint disk image.

Use a different/new image descriptor from the base descriptor, changing the name and image path. Keep the rest of the settings the same. Remember to cleanly shut down Windows and power off the node before capturing the image.

Phase 5: Build and install the Emulab client-side

This phase consists of configuring, compiling, and installing the Emulab client-side code.

  • Login as the 'root' user at the VGA console
  • Transfer over your Emulab source, definitions file and unattended setup answer file.

If not already present and up-to-date, copy your Emulab source, definitions file and unattended setup answer file to the target node. Double-check the latter two items for correctness.

  • Configure the source / build the object tree

From a Cygwin command prompt, do:

mkdir /var/tmp/client-obj
cd /var/tmp/client-obj
<path_to_your_emulab_src>/clientside/configure --with-TBDEFS=<path_to_your_defs_file>
make
make client-install
  • Hook in the Emulab services.

From a Cygwin command prompt, run:

/usr/local/etc/emulab/hookservices

Type in your same root password again when prompted.

Phase 6: Validate the client-side and make a "Release Candidate" image

  • Reboot the target node and validate that the Emulab client-side started up.

It takes a minute or so after boot for the Emulab client-side to kick in, and another minute or so before it completes. It is launched by the "EmulabStartup" service (a 'cygrunsrv' wrapped service). Once Windows is up and running again, check for Emulab processes and logs. There should be a bunch of stuff out in /var/emulab/boot, and a trail of activity in '/var/log/bootsetup.log' (looking from inside a Cygwin shell). Check /etc/passwd: You should see all of the users in your project listed. Processes such as "emulab-syncd" and "slothd" should be running ('ps -Welf | more').

  • Check for problems in the logs

Look carefully at the log files in /var/log for signs of trouble. Also check the Windows event log.

  • Test SSH

You should be able to ssh in as root from your boss server (with no password), and with your regular Emulab username from anywhere. When you log in, you should get your usual Emulab home directory and shell environment. '/share' should be the 'share' share on your 'ops' or 'fs' server, and '/proj/<your_proj>' should be the project directory for the project that you are building the image under on the target node. This mirrors what you get on a plain old Unix Emulab node. Type 'mount' to see what all is there.

  • Test RDP

Try to RDP in to the target node. Don't forget to use your Emulab Windows password, not your usual Emulab password, as discussed above. Once logged in, open a Cygwin shell. You should see your usual Emulab home directory, project directory, and the 'share' share.

  • Prepare for imaging on the target node

If everything appears to be in order, then it's time at last to make a 'release candidate' Windows image! Log out of all remote connections and go back to the VGA console. From a Cygwin command shell, run:

/usr/local/etc/emulab/prepare -s -u <path_to_your_unattended_setup_file>

Enter the same root password when prompted. The node will go through various cleanup steps and then run sysprep. The '-s' switch tells 'prepare' to run Windows sysprep after cleaning up, which will in turn shutdown Windows and power off the node after it finishes. The '-u' switch tells 'prepare' where your custom unattended setup file is located. Wait for the node to shutdown.

  • Enable 'node admin' mode for the target node.

On your ops server, run:

node_admin -n <your_target_node> on

Where <your_target_node> is the name of your target Windows node. This will ensure the node doesn't accidentally boot into Windows before an image has been captured.

  • Create the 'release candidate' image

Create an image descriptor via the web interface to hold your release candidate image. Sample config:

rc-descriptor.png

What to change in the above config:

  • Project: This should be set to match the project you are using
  • Descriptor Name: This should be set to something appropriate, such as "WIN7-RC1" for a release candidate 32-bit Windows OS
  • Node Types: Check any additional nodes that are likely to be supported in-box by this version of Windows.
    • This list can be modified later as new types are validated/added as well.
  • Node to obtain snapshot from: Name of the target node you've been building up your Windows image on.
  • Reboot Waittime: You '''MUST''' use a non-default value here to avoid timeouts during experiment setup.  A value of "1200" should do.
    • Windows takes a very long time to specialize itself after image load.

Click 'submit' to create the image descriptor and start the image capture. Note that the resulting image will be "generalized", i.e. unbound, from specific hardware by the Windows sysprep process, so can now be brought up on any node where it supports the hardware (has the requisite drivers).

  • Disable admin mode on your target node

Once the image has been successfully captured and the node has booted into the admin MFS, turn off admin mode and it should go through the mini-setup process and eventually boot to the Windows login screen. On your ops server:

node_admin <your_target_node> off

Phase 7: End-to-End Testing and Final Release

Before declaring the image golden, try swapping in a variety of experiments using it. Try it on all node types you are interested in. The following node directive is immensely useful when there are problems. It allows an experiment to swap in even if a node fails to come up properly:

tb-set-failure-action <node> "non-fatal"

Test a more complicated topology with static routing enabled and a mix of Windows and other OSes. Run linktest to validate the topology.

TODO: Add sample ns file(s)

If everything looks good, make the image global, or share it however you like. Victory!

Caveats and Known Issues

This section lists the kinks and gotchas we know about related to using and building Windows images in Emulab.
  • Users' Emulab Windows passwords must be explicitly set.
Emulab maintains a separate, clear-text password for each user account for use with Windows.  Be sure to have everyone working with Windows (users and admins) go and explicitly set their Windows password.  To do this, visit: 'My Emulab -> Edit Profile', fill in the Windows password fields, then submit the form.  If a user hasn't ever done this, they will not have a valid Samba account mapping, and many things - like home directory access on the nodes - won't work.
  • International Windows versions aren't Emulab-friendly
For better or worse, Emulab parses the output of certain command-line utilities (such as ipconfig).  The Emulab scripts can't deal with localized language strings, nor the variation in output they cause in Windows utilities that don't understand them either.  Therefore, please install and switch to the English language pack or you may run into problems (particularly in the area of experimental interface and static routing setup).

Troubleshooting

If something goes wrong in the process of building your Windows image(s), look here for guidance.

Windows setup problems

Note: You can crack open the Windows install at any point by pressing "shift-F10". This will bring up a command window.

  • Setup fails to run autonomously (aside from asking about the version of Windows to install)

Be sure you have inserted the USB memory stick, that it is formatted FAT32, and that the unattend file is in the root directory of the device, named "Autounattend.xml". If all of these items appear to be in order, then check the setup log, located here: "X:\windows\panther\setupact.log". Look toward the end of this file for clues to the issue. Did it see the USB stick? Did it see the Autounattend.xml file? Did it parse the file OK, or did it complain about something that was incorrect or missing?

  • Setup fails to install, and pops up an error message

Check "X:\windows\panther\setuperr.log" if setup is still running from the DVD, or "C:\Windows\panther\setuperr.log" if it has moved on to running from the hard disk. Also check the corresponding 'setupact.log" files in the same directories.

Sysprep problems

Look in the same log files as those mentioned in 'Windows setup problems', and also in the same files in C:\Windows\Systeme32\sysprep\panther

Base setup issues

This section deals with issues encountered while running the setup sequences in Phase 4 above. Note that these sequences should be idempotent, so can be run again if necessary to retry or gather extra debugging information.

If you run into problems executing the the first sequence of setup commands, run the script again as indicated, but add the "-debug" flag. DO NOT add the debug flag to the execution of the second command script! Debug mode tries to copy the output of all commands into the log file. The Cygwin installer produces an ENORMOUS amount of output, which will choke the setup script when it tries to copy it all in. However, if the Cygwin installation succeeded previously, but something else failed, DO add the debug flag since the second run of Cygwin setup won't produce such copious output.

Emulab Client-side Issues

  • Emulab daemons aren't running, accounts are missing, etc.

Something has gone wrong with the Emulab client bootstrap. Look in the Event Log to see if Windows tried to start the "EmulabStartup" service. Check /var/log/bootsetup.log from a Cygwin shell for clues. Is there anything interesting in /var/log/messages? Is the pubsub service running? Look through the output of:

cygrunsrv -LV

Seeking Help

If you are stumped after reviewing logs, Googling around, and searching the Emulab Admins Google forum, send along your question(s) to the emulab-admins Google Group forum. Please be very specific, erring on the side of providing too much information.

Try to answer these questions:

  • At what phase/step did things go wrong?
  • What do the log files indicate?
  • Have you deviated at all from the documented installation process? If so, how?
  • Have you modified the source code? If so, how?
  • Is there anything related in the emulab-admins Google Group archive?

Logs of interest:

X:\windows\panther\setupact.log (WinPE installer environment)
X:\windows\panther\setuperr.log (WinPE installer environment)
C:\windows\panther\setupact.log
C:\windows\panther\setuperr.log
C:\windows\system32\panther\setupact.log
C:\windows\system32\panther\setuperr.log
/var/log/bootsetup.log
/var/emulab/logs/*
/var/log/messages

Attach the relevant logs, if possible. Include the git hashtag for HEAD on the master branch (.git/refs/heads/master) for the Emulab source you are using on the client-side. Send along your Emulab definitions file and your unattended setup answer file as well.