Masterfrisbeeserver
Configuration info for the frisbee master server.
The frisbee master server runs on a known port and accepts requests to upload and download "images." Images may be specially prepared, highly-compressed imagezip .ndz images or just regular files. The master server authenticates the request and forks a frisbeed (to download) or image collector (to upload) process to handle the rest. By being aware of all activity, it can enforce aggregate resource usage limits if necessary.
Scenarios.
Emulab
A master frisbee server runs on boss. It is configured so that it uses the Emulab "config file" backend, which means it extracts configuration info on demand from the Emulab DB rather than using a literal config file that we constantly modify (and HUP the server). This may be easier said than done as we have to map DB table info into something coherent for the model below (this is why I call it an Emulab backend and not a general database backend; the latter would just use a DB to store config info in master frisbee format).
For image distribution, frisbeelauncher would go away. os_load, the elabinelab and subboss interfaces would no longer fire it off. tmcd would just return an image-id to clients via the "loadinfo" call. The frisbee client, along with inner bosses and subbosses, will request images by name using the GET interface. At elabinelab or subboss setup time, we will have to create the appropriate DB state to control which images those entities have access to.
For image collection, boss should just have to boot nodes in the admin MFS along with the current start command. That start command currently gets all the info it needs on the command line and that can remain the same. The info now includes a server and image-id rather than an NFS-accessible pathname. The create image path will need to be modified so that it ensures that the frisbee master server allows the image to be uploaded from the node in question.
Standalone
Here we allow anyone to download the frisbee software along with a CD- or dongle-boot image that contains a bootable Linux/BSD with frisbee installed. They build the server side on Linux/BSD and configure the text mfrisbee.conf file for the master server. By default the config file would have to be useful "out of the box", we'll have to figure out what that means. At the very least it would limit up/downloads to/from a certain local subtree. It might not require any authentication or be set to IP authentication of nodes for a particular subnet. We will need to have some method for creating per-user or per-node certificates, keys, or passwords for stronger authentication.
For image distribution, they would boot from the dongle/CD and initially type a command line to up/download images. One could imagine a GUI you run on your machine as well that provides a list of images to load or gives a form to fill out for upload or whatever. Something else to think about is what we can do to ease automation of large-scale disk loading. One thing would be to provide a network boot process rather than CD/dongle. Maybe the GET interface is then modified to allow a null request where the server will return a "default" image to load or some other indicator if it should not load its disk at all. The latter would tell the process to boot from disk instead of load the disk.
For image collection, it is pretty much the same: boot a CD/dongle and manually run an imagezip command, someday a GUI.
Comments
So have we really made frisbee more standalone? Not really for the initial disk-loading case, but we have made it slightly easier (or maybe faster) for saving images. More importantly though, we have made it easier to use frisbee to scalably download arbitrary files in the regular OS context. You can just fire off a frisbee command on any number of nodes and it will work; no wrapping a file as an image, picking a MC addr/port, starting a server and then running a bunch of frisbees on nodes.
Authentication
What authentication makes sense for frisbee (downloading images)? Note first of all that authentication does not imply any on-the-wire encryption or integrity protection. (You can, however, create encrypted and/or integrity protected images with imagezip which can then be transferred with frisbee.) Since frisbee uses a UDP multicast protocol, it does not prevent unauthorized parties from downloading an image; i.e., joining the multicast group. So the best we do here, without modifying the frisbee protocol, is to have the client/server check the IP address of the sender and filter based on that. So the only meaningful per-user authentication or key- or certificate-based authentication we can do for download is in the master server. This would allow us to do something like a secure transfer of the image encryption or signing key. It also allows us to limit the number of frisbeed processes that get spawned for feeding images (since only authenticated users/nodes would be able to do a GET which in turn fires off the frisbeed process) and thus it can help control resource consumption on the server.
For uploading images, we could do a lot better since we are just going to use a unicast TCP stream. So we can establish an SSH/SSL connection if desired.
This asymmetry bothers me, but I don't know what to say.
Anyway, part of the master server configuration will be the type(s) of authentication it will support. As we have seen, these might be different for upload vs. download. The simplest type is "none" which will allow any request to be serviced. The server will run with limited privilege of course to ensure that frisbeed doesn't offer up or clobber any protected system files. "IP" will validate based on the sending IP address of the request. This configuration is suitable for environments such as Emulab where host-based authentication is what we need and IP-spoofing is prevented. "SSL" will allow host or user based authentication. I suppose this could allow per-image authentication as well. We could have a "password" type requiring requests to provide a password to allow per user/host/image access.
For Emulab, we just need "IP". For standalone, we would also want a "none" and maybe some form of key/cert/password auth along with "IP".
Image IDs
What exactly is an image ID? The only criteria here is that it be globally unique. We could do a standard UUID, but I would prefer to keep it to something fairly human readable. So for now just an ASCII string. Note that these are just used by the client to name the image at the server. The server will map that ID into a UUID or path or Venti hash or whatever.
For Emulab an ID will just be a "pid/eid" style string. For standalone, it is also just a string, with it being up to the user to figure out a unique naming convention.
Request types
GET <image-id> <options>
Determine if the caller can access the image identified by <image-id> with the desired <options>. If so a frisbeed is spawned and the necessary information returned for the caller to use it. <options> specifies desired characteristics, which the server can grant or not:
methods=<multicast|unicast> # desired xfer method(s) status=<0|1> # just return access status for image bandwidth=<rate> # desired max download rate priority=<high|low> # random QoS category Returns: error=<code> # 0 on success, failure code otherwise method=<multicast|unicast> # xfer method chosen address=<addr>:<port> # mcast/ucast IP info bandwidth=<rate> # max download rate priority=<high|low> # QoS setting frisbee-options=<string> # "ideal" settings; e.g., readahead value signing-key=<key> # signing key for integrity-protected images encryption-key=<key> # encryption key for confidential images isrunning=<0|1> # is a daemon running (for statusonly call)To use this, the frisbee client will have a command line option where an image-id is specified rather than the current address/port info. It will use the GET call to find out the addr/port information from its master server. This eliminates the need for an out-of-band channel for communicating this info.
For return values we must have method, address and maybe frisbee-options so I can mess with client-side tuning. authentication/encryption will be important when I get the "secure frisbee" changes merged in.
PUT <image-id> <options>
Upload an image to be identified by <image-id>. Options:
create=<0|1> # create if it doesn't already exist size=<bytes> # size of the image in bytes statusonly=1 # just return whether we can upload the image signing-key=<key> # signing key for integrity-protected images encryption-key=<key> # encryption key for confidential images Returns: address=<addr>:<port> # ucast IP info address=current # use the current connection size=<bytes> # max size of the image that will be accepted signing-key=<key> # signing key for integrity-protected images encryption-key=<key> # encryption key for confidential imagesNote that there are keys in both directions. One could imagine that the creator of an image might want to choose the keys used, but I am not sure where these would be put. I view the config file as RO to the master server (even if it is a DB) so I don't see it changing that. It could store them in files alongside the image I suppose. Alternatively, the user could let the master server choose them and they would be returned as shown. Still have to store them somewhere though...needs more thought.
To use this command, imagezip will have an option, taking a server name and image-id, allowing it to upload its image rather than storing it locally. After getting back the reply, it will either open a new connection or start sending the image data on the current connection.
We might also want some sort of "imageupload" that will upload an existing image rather than the imagezip style create-and-stream-it.
LIST
Return a list of images that the caller (user or node) has access to. Includes information like: sizes of images, current IP info for running servers, and max download bandwidth.
Config stuff
File/image section
image <image-id> path="/path/to/file" # location of image auth-key=<key> # signing key for chunks enc-key=<key> # encryption key for data get-options=<string> # additional options for frisbeed (timeouts, etc.) put-options=<string> # additional options for the uploader (max size, etc.)
"get-options" is a hack to address misc. things like server timeouts and multicast keepalives. They could each have their own option name, but really, who cares.
Authentication section
authtype <node|user> <upload|download|updown> <none|SSL|IP|any>
Global option to specify how authentication is done. Could be multiple lines to control authentication by node or user. There can be different types for upload vs. download of images. Type "none" will allow matching only against the "any" <host-id> or the "anybody" <user-id> in the authentication rules below. "SSL" will accept only SSL connections with certs for users or nodes. "IP" will tell the master to only match host rules which match the sender of the request (useful for Emulab where IP-spoofing is prevented).
host <host-id> <permissions> <image-id> ...Node <host-id> has the specified permissions on the indicated images. <host-id> can be a name or an addr/mask. Permissions indicate which of the above operations is permitted (or ALL).
user <user-id> <permissions> <image-id> ...User <user-id> has the specified permissions on the indicated images. Permissions indicate which of the above operations is permitted (or ALL).
Resource use section
host <host-id> image-size=<bytes> # max bytes for an upload of any one image from host total-size=<bytes> # max bytes for all uploads from host image-rate=<bytes/s> # max xfer rate for any one image from host total-rate=<bytes/s> # max aggregate xfer rate for host
user <user-id> image-size=<bytes> # max bytes for an upload of any one image from user total-size=<bytes> # max bytes for all uploads from user image-rate=<bytes/s> # max xfer rate for any one image from user total-rate=<bytes/s> # max aggregate xfer rate for user
Example configurations
(I'm using a Bacula-like syntax in the following, but we might want to go with a more XML-ish syntax like Apache's.)
Emulab
For Emulab, the real configuration would be in the Emulab DB, but here is what it would map to as a standalone configuration. Emulab projects (groups) control what images can be placed on what node. Users create experiments which belong to a particular project, and those experiments contain nodes. Thus at any time, a particular node has access to a particular set of images. Since nodes have fixed control network IP addresses, at the end of the day, image access can be controlled by IP address.
So in theory, the Emulab master frisbee config file is a set of "host" authentication lines which which contain the current set of standard and project-specific images that can be loaded on that host.
Section Auth {
# authenticate node downloads by IP address
authtype node download IP
# pc401/402 are in experiment testbed/foo; can load testbed, standard images
host 155.98.37.1 GET testbed/MIKES-IMAGE ... emulab-ops/FBSD81-STD ...
host 155.98.37.2 GET testbed/MIKES-IMAGE ... emulab-ops/FBSD81-STD ...
# pc403 is in experiment emulab-ops/reloading; can load standard images
host 155.98.37.3 GET emulab-ops/FBSD81-STD ...
# pc404 and beyond are free; cannot load any image
host 155.98.37.4 GET none
host 155.98.37.5 GET none
...
}
Clearly, if we were going to write this in a text format, we would want some form of aliasing and/or wildcarding. For example an IP alias that would allow you do define a name to associate with a group of IP addresses:
ipalias "testbed/foo" 155.98.37.1 155.98.37.2
ipalias "testbed/reloading" 155.98.37.3
and ditto for images:
# globs...
imagealias "proj-testbed" testbed/*
imagealias "standard" emulab-ops/*
# ...or REs
imagealias "proj-testbed" "^testbed/[^/]+$"
imagealias "standard" "^emulab-ops/[^/]+$"
and some sort of catch all category for nodes not explicitly mentioned:
Section Auth {
# authenticate node downloads by IP address
authtype node download IP
# nodes default to free, not allowing downloads
host * GET none
# pc401/402 are in experiment testbed/foo; can load testbed, standard images
host testbed/foo GET proj-testbed standard
# pc403 is in experiment emulab-ops/reloading; can load standard images
host testbed/reloading GET standard
}
For uploading images we would again limit by IP, allowing nodes to only update images for their project:
Section Auth {
# authenticate node downloads/uploads by IP address
authtype node updown IP
# nodes default to free, not allowing uploads or downloads
host * ALL none
# pc401/402 are in experiment testbed/foo:
# can load testbed, standard images
# can update testbed images
host testbed/foo GET proj-testbed standard
host testbed/foo PUT proj-testbed
# pc403 is in experiment emulab-ops/reloading; can load standard images
host testbed/reloading GET standard
host testbed/reloading PUT none
}
For images, we will name them with IDs like "<pid>/<imageid>" as we see above. There again will (conceptually) be one line per image:
Section Images {
image "emulab-ops/FBSD81-STD" {
# found in the standard image directory
path "/usr/testbed/images/FBSD81-STD.ndz"
# keep inactive frisbeed alive for 30 minutes after last request
get-options "-T 1800"
}
...
image "testbed/MIKES-IMAGE" {
# found in the testbed project images directory
path "/proj/testbed/images/MIKES-IMAGE.ndz"
# short inactive period, limit to 54Mb/sec
get-options "-T 300 -W 54000000"
}
...
}
And of course, if we were really doing it this way we would want some syntactic sugar to keep things short, maybe Perl RE based:
Section Images {
# system images are found in /usr/testbed/images
image "^emulab-ops/(.+)$" {
path "/usr/testbed/images/$1.ndz"
get-options "-T 1800"
}
# testbed/<foo> images are found in /proj/testbed/ as "foo.ndz"
image "^testbed/(.+)$" {
path "/proj/testbed/images/$1.ndz"
get-options "-T 300 -W 54000000"
}
...
}
But what about uploading of new images? As shown, the imageid for a new image would have to already exist in order to allow upload. In reality, the Emulab config which is resident in the DB would do just that. We would dynamically create the imageid before firing off the process that would do the upload.
But this is not particularly satisfying. Even in Emulab we might want users to be able to upload arbitrary experiment-related files via this mechanism. So maybe augmenting with:
Section Auth {
...
host testbed/foo ALL "/proj/testbed/exp/foo/expfiles/*"
...
}
Section Images {
...
# testbed/<foo> user files can be put in a special directory
image "^(/proj/testbed/exp/foo/expfiles/.+)$" {
path "$1"
}
...
}
So users can create "images" that have an id that just happens to correspond to their special location in the FS. So someone would do something like:
imageupload /proj/testbed/exp/foo/expfiles/mikestarball.tar.gz
to upload, or:
frisbee -N /proj/testbed/exp/foo/expfiles/mikestarball.tar.gz /tmp/foo.tar.gz
to download.
Note that it is superficially weird to authenticate based on node IP address when allowing upload of experiment (user) files. This is okay in Emulab because at any time a node is allocated to a particular experiment (user) and we dynamically change the config when the mapping changes.
I'm going to ignore the "Resource Usage" section for now since we can get away without it, I haven't thought much about it, and I don't know how to implement some of the features (e.g., aggregate BW limiting). We can use the images section "get-options" option to specify everything we do today.
Standalone:
So what would a simple standalone config look like? How about:
Section Auth {
# authenticate node downloads by IP address
authtype node download IP
# from any node on our network
host 10.1.1.0/24 GET any
# new images are created on my special machine
host 10.1.1.200 PUT any
}
Section Images {
# All images come from our image repo
image "^([^/]+)$" {
path="/images/$1.ndz"
}
}
This would allow image download to nodes on a particular subnet, and uploads from a particular "image creation" machine.
Current Emulab Implementation (11/24/2010)
These commands are implemented as Frisbee (binary) packet types in network.h. Currently, only GETREQUEST and GETREPLY packets are implemented:
typedef struct {
int32_t type;
union {
struct {
uint8_t methods;
uint8_t status;
uint16_t idlen;
char imageid[MS_MAXIDLEN];
} __attribute__((__packed__)) getrequest;
struct {
uint8_t method;
uint8_t isrunning;
uint16_t error;
in_addr_t addr;
in_port_t port;
} __attribute__((__packed__)) getreply;
} body;
} MasterMsg_t;
An imageid can be up to 256 bytes (large enough to hold an ASCII-encoded SHA-1024 hash). Right now the only supported method is multicast.
The client uses the -S option to specify the IP address of the master server and the -F option to specify an imageid. If no error is returned, the image is downloaded using the returned addr/port. If -Q is used in place of -F, then the client makes a "status only" call getting back info about whether the named image is accessible to the client and whether a server is currently running.
On the server side, the new master server (mserver.c) has an Emulab configuration "backend" that supports host-based authentication. The IP address of the caller is mapped to a node_id/pid/gid/eid combo that is used to determine access. On a request, the specified imageid is treated either as a pathname (if it starts with '/') or an image identifier of the form "<pid>/<imagename>". If it is a pathname, we check to make sure that pathname (after running through "realpath") is contained in one of the directories accessible to that node in its current experiment context; i.e., /share, /proj/<pid>, /groups/<pid>/<gid>, or /users/<swapper-uid>. If it is an image identifier, the DB is queried to ensure that access is allowed to that image; i.e., it must be "global" or in the appropriate project/group.
The master server forks a frisbeed for each valid request, if one is not already running. The multicast address selection is still based on the emulab_indicies.frisbee_index field, but the address/port/server info is no longer stored in the frisbee_blobs table (frisbee_pid, load_address, load_busy are not set).
Note that this is not yet integrated. Further work is required to eliminate frisbeelauncher.
Hierarchy (inner bosses, subbosses).
I have not thought this through yet.
Interaction with Venti and a new imaging subsystem.
Think about it. For Venti, this is a backend storage issue; i.e., having a modular interface for fetching and storing image data and having some URI-like naming convention for files so that we can choose the correct storage interface. How this all integrates with the bigger picture of a new imaging subsystem is still TBD.
Interaction with the trusted disk-loading system (TDLS).
Think about it. This might affect authentication if we wanted to use the TPM-protected identity key for a node. Also, key exchange (for the image signing and encryption keys) will need to be done using the node key.