Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Wiki / Snmpitinternals

Snmpitinternals

Introduction

Some notes on the design and operation of snmpit Robert Ricci <ricci@cs.utah.edu>, Keith Sklower <sklower@cs.Berkeley.EDU>. And notes on the new switch support of snmpit, by Weibin Sun <wbsun@cs.utah.edu>

Introduction

This document describes the design and implementation of snmpit, the command line tool in Emulab to control the switches for experiment VLAN, link and tunnel manipulation. There is also a list of functions that should be implemented to support new switch.

File organization

snmpit - The command line tool. Contains most of the database accesses, does permissions checking, formats output, and figures out which device-specific backends it needs to invoke. Most emulab-specific knowledge is embedded in snmpit. If you're going to add another switch backend to snmpit, the only part of snmpit itself that you should have to change is the part that loads the device library - search for 'cisco' in the file.

snmpit_lib.pm - functions useful to snmpit and common to multiple device backends. The most important functions to be aware of in here are ReadTranlationTable() and the snmpitGetWarn() and snmpitGetFatal() functions. The latter two wrap SNMP commands, retry in case of timeout, and send mail to the site's testbed-ops list if they fail. The first simply warns the user and continues execution, whereas the Fatal() version exit()s.

snmpit_stack.pm - Contains the knowledge required to handle a collection of switches which share a common set of VLANs. Does not actually do any SNMP itself, but has the knowledge of how to deal with multiple switches. For example, it knows that (in some configurations) one switch acts as a 'VLAN server', and in order to create a VLAN, you just have to talk to it. But, to get a list of ports in VLANs, you have to talk to all of the switches. Also has the job of aggregating information from switches - ie. doing a listVlans() on the stack does a listVlans() on all individual switches and then collates the results. snmpit itself calls into the stack module, which creates a snmpit_<switchtype> module for each switch, and calls into that to do the actual work.

snmpit_cisco_stack.pm - Contains the knowledge required to handle a collection of Cisco switches which share a common set of VLANs. There are optimization that could be done for collections in which all the switches are cisco switches, but currently both emulab and DETER using snmpit_stack.pm instead.

snmpit_cisco.pm - Contains the actual SNMP commands to deal with Cisco switches, and deals with error checking, retries, etc. Writing versions of this for other switches will be the hardest part of porting snmpit, because it has a lot of knowledge about the quirkiness of Cisco's SNMP implementation.

snmpit_nortel.pm - Contains the specific SNMP for Nortel 55[123]0 class switches, and deals with error checking, retries, etc.

snmpit_foundry.pm - Contains the specific SNMP for Foundry 9604 and 1500 class switches, and deals with error checking, retries, etc.

snmpit_intel_stack.pm - Similar to snmpit_cisco_stack.pm . We haven't used it in quite some time, so it's quite possibly bitrotted.

snmpit_intel.pm - Ditto - like snmpit_cisco.pm .

snmpit_apc.pm - For controlling APC-brand SNMP-controllable power controllers. Don't be fooled by it's name - it's not actually part of snmpit anymore, it just used to be. It's now used only by the 'power' command, and you can ignore it.

snmpit_apcon.pm - For controlling Apcon 2000 class physical layer switches via SSH. It has nothing related to SNMP and the name is for compatibility only. If you want to add new switch support that uses different approach rather than SNMP, this is the file you should look at. It uses the Perl Expect module to connect to the switch via SSH.

snmpit_hp.pm - Contains the actual SNMP commands for HP Procurve class switches. Till now(Sep, 2010), this is the only file containing the actual logic to do Openflow settings on switch via SNMP. The Openflow functions in other device packages are just dummy functions that do nothing. If your new switch supports Openflow, you might want to read through the functions having 'Openflow' in their names.

Design philosophy

One of the key issues is that we trust switch state over database state. Thus, when we go to remove a VLAN, we do not trust the database to list all the ports in that VLAN for us - we get the list of member ports from the switch. The idea here is twofold - first, we don't want to get too confused if people have been manually manipulating VLANs on the switch, which does happen from time to time, and usually happens for good reasons. Second, we don't want to have to worry too much about getting out of sync, which can be a huge mess - for example, if the database has to be restored from a backup, or if someone messes around with the database by hand.

On the other hand, we don't trust the switches _too_ much - we usually verify that set operations have succeeded before reporting success. ie. we'll set the port speed to 100Mbps, then check the speed to make sure the change actually took place. It's been our experience that, since we're pushing this stuff far harder than it was intended (really, how many sites have made hundreds of thousands of VLANs?), we do occasionally hit bugs in the switches.

Many of the API choices were made in order to enable high performance in the backends. For example, we can supply a list of ports to setPortVlan() (in the stack module), rather than just a single port, because it may be faster to affect multiple ports at once than to do them serially. Whether or not you choose to exploit this bit of API design is up to you.

Switch stacks

snmpit has a concept of stacks of switches. These are a set of switches on which we can create a VLAN that spans potentially all switches. Thus, they are connected by trunk links. Right now, snmpit does not support creating a VLAN across multiple stacks (that would go against the definition of a stack), but it does not stacks that contain more than one type of switch.

Probably, all of your experimental-net switches will be in a single stack.

Stacks are, by convention, sometimes named after their leader. In a Cisco stack, the leader is the one you talk to in order to create VLANs, etc, via VTP; however in a generic stack, the two are usual named "Control" and "Experiment", where the "Control" stack is used for setting up firewall vlans and the "Experiment" stack is used for normal links and lans described in .ns files.

New switch backends

Essentially, what you need to do to port snmpit to a new switch vendor is make a new module that exports the same API as snmpit_stack. Let's assume the new switch is called abc switch and the device package is snmpit_abc.pm. The following list shows the functions you need implement in this module.

For your convenience, I attached a startup snmpit_abc.pm so that you can simply fill your logic in the empty functions. Some default codes are also in it and you can change freely if you are in different case.

  • sub new($classname,$devicename,$debuglevel,$community)
    • Description: return a new object which is blessed into the snmpit_abc class. Common operations inside are create connection to switch, initialize settings from db.
    • Parameters:
      • $classname: should be 'snmpit_abc', at the end of this function, we 'bless' the object to be 'classname' type.
      • $devicename: the switch device name, can be used for connection in SNMP or SSH or Telnet, etc.
      • $debuglevel: whether we display the debug info or not, and what kind of info should be displayed, see debug().
      • $community: for SNMP community setting, if your connection is not SNMP based, this is a good place to put the password.
    • Return: the $classname typed object if success, or undef on failure.
  • sub portControl($self, $command, @ports)
    • Description: set port rate, port link type and/or enable/disable ports
    • Parameters:
      • $self: this object.
      • $command: The port property value, typically, it is one of the following values:
        • "enable": enable port
        • "disable": disable port
        • "1000mbit": set the port rate to be 1000Mbps
        • "100mbit": set the port rate to be 100Mbps
        • "10mbit": set the port rate to be 10Mbps
        • "full": set the link on the port to be full-duplex
        • "half": set the link on the port to be half-duplex
        • "auto": set the link on the port to be auto mode
      • @ports: the array containing the port-numbers of ports we want to manipulate.
    • Return:
      • 0 on success.
      • number of failed ports on failure.
      • -1 if the operation is unsupported.
  • sub findVlans($self, @vlan_ids)
    • Description: find the corresponding VLAN numbers by VLAN ids
    • Parameters:
      • $self: this object.
      • @vlan_ids: ids of VLANs to be found.
    • Return: a hashmap: VLAN id => VLAN number, the VLAN number will be undef if VLAN is not found. If no VLAN id is given, returns mappings for the entire switch.
  • sub findVlan($self, $vlan_id, $no_retry)
    • Description: find the corresponding VLAN number by VLAN id
    • Parameters:
      • $self: this object.
      • $vlan_id: id of the VLAN to be found.
      • $no_retry: Optional, whether retry or not on failure. Can be 0 or 1.
    • Return: VLAN number, the VLAN number will be undef if VLAN is not found.
  • sub vlanNumberExists($self, $vlan_number)
    • Description: Check to see if the given VLAN number exists on the switch
    • Parameters:
      • $self: this object.
      • $vlan_number: number of the VLAN to be found.
    • Return: 1 if found, otherwise 0.
  • sub createVlan($self, $vlan_id, $vlan_number)
    • Description: create a VLAN on switch with the given VLAN id and its number. (Note-1: in the case of the Foundry switch, you cannot create a vlan with no elements, so the module cashes the information, fibs about having does it, and looks it up when ports are to be added.) (Note-2: on Apcon physical layer switch it has no idea about VLAN. So snmpit uses port name as the VLAN name it belongs to. No VLAN can be created with no ports. the module will save the empty VLANs and print a message when unloaded to warn about the empty VLANs to be deleted.)
    • Parameters:
      • $self: this object.
      • $vlan_id: id of the VLAN to be created.
      • $vlan_number: VLAN number of the VLAN to be created.
    • Return: the new VLAN number on success or 0 on failure. Note the new VLAN number may be different from the given number if it's not available on the switch.
  • sub setPortVlan($self, $vlan_number, @ports)
    • Description: put the given ports into the given VLAN which is identified by its VLAN number
    • Parameters:
      • $self: this object.
      • $vlan_number: the VLAN number of the given VLAN.
      • @ports: the array of ports to be added into the VLAN. (Note: in the case of Apcon physical layer switch, only as many as two ports can be added into a VLAN.)
    • Return: 0 on success or number of failed ports on failure.
  • sub delPortVlan($self, $vlan_number, @ports)
    • Description: remove&disable the given ports from the given VLAN which is identified by its VLAN number
    • Parameters:
      • $self: this object.
      • $vlan_number: the VLAN number of the given VLAN.
      • @ports: the array of ports to be removed from the VLAN
    • Return: 0 on success or number of failed ports on failure.
  • sub removePortsFromVlan($self, @vlan_numbers)
    • Description: remove all ports from the given VLANs which are identified by their VLAN numbers
    • Parameters:
      • $self: this object.
      • @vlan_number: the array of VLAN numbers of the given VLANs.
    • Return:
      • 0 on success.
      • number of failed ports on failure.
  • sub removeSomePortsFromVlan($self, $vlan_number, @ports)
    • Description: remove the given ports from the given VLAN which is identified by its VLAN number
    • Parameters:
      • $self: this object.
      • $vlan_number: the VLAN number of the given VLAN.
      • @ports: array of ports to be removed.
    • Return:
      • 0 on success.
      • number of failed ports on failure.
  • sub removeVlan($self, @vlan_numbers)
    • Description: remove the given VLAN. This includes remove all ports and clear the VLAN record on switch.
    • Parameters:
      • $self: this object.
      • @vlan_number:s the VLAN numbers of the given VLANs to be removed.
    • Return:
      • 1 on success.
      • 0 on failure
  • sub vlanHasPorts($self, $vlan_number)
    • Description: check if the VLAN has any ports in it
    • Parameters:
      • $self: this object.
      • $vlan_number: the VLAN number of the given VLAN.
    • Return:
      • 1 if VLAN has any ports.
      • 0 if VLAN has no port or doesn't exist.
  • sub listVlans($self)
    • Description: list all VLANs on the switch
    • Parameters:
      • $self: this object.
    • Return: list of array of this format: [$vlan_name, $vlan_id, @member_ports]
      • $vlan_name: meaning fule VLAN name on switch, can be $vlan_id if not supported.
      • $vlan_id: the VLAN id.
      • @member_ports: array of ports in the VLAN
  • sub listPorts($self)
    • Description: list all ports on a switch device.
    • Parameters:
      • $self: this object.
    • Return: list of array, whose format is: [$port,$abled,$link,$speed,$duplex].
      • $port: the port number in node:port format.
      • $able: yes/no in string, whether the port is enabled or not.
      • $link: up/down in string, whether the link is up or down.
      • $speed: the port rate part of $link in string, it is "1000Mbps" in the above example. Some switches support "neg" which means "negotiable", we use "auto" if it is "neg".
      • $duplex: the duplex mode of $link in string, it is "full" in the above example. Some switches support "auto" and we like it.
  • sub getStats($self)
    • Description: get statistics info for ports on the switch
    • Parameters:
      • $self: this object.
    • Return: this function is largely device independent, but uses snmp to retrieve a list of standard SNMP quantities from the non-proprietary interface MIB, (octets in, out, unicast packets in, out, etc.) It is put in the snmpit_device() file since there might be a device depending mapping between modport and ifIndex, and it does use snmp. The value is not really documented, but only is relevant if you change portstats.in or snmpit.in.
  • sub getChannelIfIndex($self, @ports)
    • Description: get the ifindex, interface index, for and EtherChannel (trunk). This is about the trunking mode of switch. I have a little idea about trunking. May need other guys to write the following explanations...
    • Parameters:
      • $self: this object.
      • @ports: array of ports to be queried.
    • Return: undef if more than one port is given, and no channel is found an ifindex if a channel is found and/or only one port is given.
  • sub setVlansOnTrunk($self, $modport, $value, @vlan_numbers)
    • Description: enable or disable ports on a trunk. This is about the trunking mode of switch.
    • Parameters:
      • $self: this object.
      • $modport: module.port of the trunk to operate on.
      • $value: 0 to disallow the VLAN on the trunk, 1 to allow it
      • @vlan_numbers: array of VLAN numbers to be enabled/disabled.
    • Return: 1 on success, 0 otherwise.
  • sub enablePortTrunking2($self, $modport, $nativevlan, $equaltrunking)
    • Description: enable trunking on a port.
    • Parameters:
      • $self: this object.
      • $modport: module.port of the trunk to operate on.
      • $nativevlan: VLAN number of the native VLAN for this trunk
      • $equaltrunk: whether don't do dual mode & tag PVID or not. 0 or 1.
    • Return: 1 on success, 0 otherwise.
  • sub disablePortTrunking($self, $modport)
    • Description: disable trunking on a port.
    • Parameters:
      • $self: this object.
      • $modport: module.port of the trunk to operate on.
    • Return: 1 on success, 0 otherwise.
  • sub enableOpenflow($self, $vlan_id)
    • Description: enable Openflow for a VLAN on this switch
    • Parameters:
      • $self: this object.
      • $vlan_id: the VLAN id
    • Return: # of errors.
  • sub disableOpenflow($self, $vlan_id)
    • Description: disable Openflow for a VLAN on this switch
    • Parameters:
      • $self: this object.
      • $vlan_id: the VLAN id
    • Return: # of errors.
  • sub setOpenflowController($self, $vlan_id, $controller)
    • Description: set Openflow controller for a VLAN
    • Parameters:
      • $self: this object.
      • $vlan_id: the VLAN id
      • $listener: the controller connection string, typically in such format: "tcp:ip:port#".
    • Return: # of errors.
  • sub setOpenflowListener($self, $vlan_id, $listener)
    • Description: set Openflow listener for a VLAN
    • Parameters:
      • $self: this object.
      • $vlan_id: the VLAN id
      • $listener: the listener connection string, typically in such format: "ptcp:port#".
    • Return: # of errors.
  • sub getUsedOpenflowListenerPorts($self)
    • Description: get the used ports by existing Openflow listeners on the switch
    • Parameters:
      • $self: this object.
    • Return: # of errors.
  • sub isOpenflowSupported($self)
    • Description: check if Openflow is supported on this switch.
    • Parameters:
      • $self: this object.
    • Return: 1 when supported, 0 otherwise.
  • sub debug($self, $message, $level)
    • Description: Optional Never called by outside code but helpful inside the module. prints out a debugging message, but only if debugging is on. If a level is given, the debuglevel must be >= that level for the message to print. If the level is omitted, 1 is assumed.
    • Parameters:
      • $self: this object.
      • $message: debug msg.
      • $level: Optional message level.
    • Return: void
  • sub lock
    • Description: Optional Never called by outside code but helpful inside the module. acquire the lock in the scope of the module.
    • Parameters:
      • $self: this object.
    • Return: void
  • sub unlock
    • Description: Optional Never called by outside code but helpful inside the module. release the lock.
    • Parameters:
      • $self: this object.
    • Return: void

Looking at snmpit_cisco.pm to figure out the basics of switch configuration with SNMP is not a bad idea, but keep in mind that there are several things which make this module very complicated.

First, it supports two different switch operating systems, IOS and CatOS, so there are some special cases for each.

Second, it supports some wacky features that you (hopefully) won't have to worry about, like 'private VLANs'. This, in particular, leads to some complex cases in setPortVlan() and createVlan().

Third, in different MIBs, ports are 'addressed' differently. In standardized MIBs, ports tend to be referred to by an 'ifIndex', which is just an integer. Cisco likes to refer to ports as a 'module.port' (at least, in CatOS), since this is the 'native' way to name ports on their modular switches. So, we have to convert back and forth between the two formats. So, keep in mind that operations for which I convert the ports into $PORT_FORMAT_IFINDEX are ones that _might_ be supported in your switch.

You can also look at snmpit_intel.pm for examples if you wish, but keep in mind it's not actively maintained.

VLAN IDs and VLAN numbers

This part causes a lot of confusion, sorry. There are two different ways we refer to VLANs, which can get confusing, 'cause they're both integers. A VLAN ID is a made-up number that we use to identify a VLAN - it's an auto-increment value in the vlans table. The actual VLAN on the switch also has a number associated with it.

Let's give an example. I create an experiment called testbed/threenodes, and I put the three members, Dusty, Lucky, and Ned, into a LAN called 'amigos'. Let's say this gets an ID of 314 in the VLAN table.

Now I need to create this VLAN on the switches. The snmpit_cisco_stack module goes out and finds the list of VLAN _numbers_ that already exist on the switch. This is the list you'd get from, say, a 'show vlan' command on a Cisco switch. Let's say this list of VLANs is 1,2,3,4,5,7,9,10,12. The module finds the first unused number (6), and uses that for the new VLAN. Note that there are holes in the VLAN list, presumably from VLANs that were previously created and then deleted.

So now we have a VLAN with ID 314 and number 6. We have to store this mapping somewhere so that future invocations of snmpit will be able to find the VLAN. For Cisco's we do this by setting the VLAN's 'name' field to the ID. So now we have a VLAN with number 6 and the name '314'. Got it? This mapping may have to be stored somewhere else, like in the database, for other switches, if you can't set names for VLANs.

The upshot is, be careful as you're looking at the API to distinguish between functions that take VLAN names and ones that take VLAN numbers. It should be pretty clear from the comments and/or variable names. Send me mail if you find any that aren't clear.

An exception is the Apcon physical layer switches that have no idea about VLAN at all. In the snmpit_apcon.pm module, we use the port name as both VLAN ID and number. So the VLAN ID and VLAN number are the same on Apcon switch.

Stack options

The switch_stack types table holds options for each stack. They are:
stack_type:		Used by snmpit to figure out which backend module to
			load
supports_private:	Cisco-specific, don't worry about it
single_domain:		Whether all switches in the stack share a VLAN domain
			(such as using VTP on Ciscos), or whether you need to
			talk to each switch individually to create VLANs. You
			can decide if it makes sense to implement this option
			for your switches or not.
snmp_community:		The SNMP community string that will be used for
			read/write access to the switches. You should support
			this option, and default to 'public' if no community
			is given. There is an exception that the Apcon physical
                        switch module uses this field to store the password to
                        login to the switch via SSH. So for modules that don't
                        use SNMP to control the switch, this may be a good 
                        place to store your login information.
min_vlan:		The minimum VLAN number (remember, not ID) that your
			module is allowed to create on the switch. It would be
			good to implement this if possible. This can be useful
			if the switch is being used for more than one purpose
			- ie. someone else could be creating VLANs in the
			range 1-500, and your Emulab could be using VLANs 501
			- 1000.
max_vlan:		Like min_vlan, silly!

Misc. things to be aware of

We want to disable ports that are not currently part of an experiment. So, when we tear down a VLAN, we have to do something to the ports that were previously in it. On Intel switches, you can actually have ports that are not in any VLANs, so we disable them and remove them from their VLAN. For Ciscos, however, ports are always in a VLAN - so we move them into VLAN 1, and set them to 'disabled'.

In snmpit_cisco, we have two ways to create VLANs on switches - one is using VTP, in which we create VLANs on a 'leader' and the switches do the job of getting it created everywhere. The other scheme has some interesting properties. Depending on what your switches support, you may have to deal with some of the same issues. In it, we have to talk to all switches to create VLANs. We have made a decision that, though poor from a performance perspective, helps consistency. We create the VLAN on _all_ switches, regardless of which ones actually have ports in it. This way, we do not have to deal with issues that come from different switches having differing sets of VLANs. Our locking protocol is such that when we create a VLAN, we do so on the switches in lexicographically sorted order - when we delete a VLAN, we do it in reverse lexicographic order. This way, we don't have different concurrent instances of snmpit pick the same VLAN name because they are getting VLAN lists from different switches.

Where to learn about new switch vendors and SNMP

There are three ways in which we've found out which SNMP commands we needed to invoke, in what order, to get this stuff to work.

Originally, we started by using the vendor's own SNMP configuration tools and tcpdump-ing them. This can end up being a bit confusing, but it's the most reliable source for discoving the switches' quirks.

We've also grabbed all the MIB files from the vendor (Cisco's are at: ftp://ftp.cisco.com/pub/mibs/v2), and looked through the OID names and comments. Luckily, the comments in Cisco MIB files are pretty good.

Finally, some vendors may provide documentation on how to perform some actions. I wouldn't count on this one - Cisco's documentation, for example, is very lacking in this area. I actually had a Cisco support person tell me that it was not possible to do through SNMP something we'd been doing for years.

You can, of course, try out the OIDs we use in snmpit_cisco and snmpit_intel - some of them may be supported on your switch.

switchmac

switchmac is a script, used at installation time, to print out all the MAC adresses a switch has learned, and what ports they're on. The idea is to eliminate the need for the testbed administrators to manually enter which nodes are connected to which ports on which switches.

The stuff in this should be pretty standard - ie. this script has support for Intel, Nortel, and Cisco switches, and I suspect that one of these should work pretty similarly to whatever switch you're trying to port it to.

In your switch vendor's documentation, take a look for the dot1dTpFdbTable in the BRIDGE-MIB (which is a standard one.)

The main ickiness in this script is something called 'Community String Indexing' - see, the MIB is designed for switches that store one MAC table that is VLAN-independant (ie. you can't use the same MAC in two different VLANs at once.) Cisco keeps a MAC table per-VLAN. So, the way they get around this is to change the community string you use to get access to the MAC table for each VLAN. So, if your community string is 'public', and you want to look at the MAC table for VLAN 42, you talk to the switch with the community string 'public@42'. Terrible. I don't know if other switch vendors use this same hack, or if they will have some other hack of their own.

Examples

* Creating custom support for legacy switches: http://www.benninger.ca/?p=38