Setting up Cisco Switches

This document contains some configuration guidelines that we (Utah) have found useful to improve the performance of our Cisco switches.

All commands given are to be entered at the (enable) prompt on your Cisco switches. They are for CatOS -- switches that run IOS will likely have alternate versions of these commands, most of which can be found at the bottom.

<ports> means a list of ports, which on the CatOS command line can include lists and ranges, such as "3/1,3/2" or "3/1-48" or "3/1-48,4/1-48,5/1-48".

At some point, our network troubleshooting guide might be useful.

Allowing ports to come up quicker

This is useful on both the experimental and control networks:

set spantree portfast <ports> enable

Use this on all ports that are directly connected to nodes, servers, power controllers - anything this is not another switch. Normally, the switch waits a while (several seconds) when a port first comes up before forwarding traffic from the port - it does so to prevent loops in the switch topology. The main place you will see the benefit of this is on the control net - with portfast disabled, the first several DHCP packets sent by booting nodes will be dropped, causing the DHCP to take much longer than necessary.

Reducing stray traffic

Disable spanning tree protocol (STP). If enabled, STP sends out packets approximately every two seconds on every port. You can disable it on all VLANs with the command

set spantree disable all

There are two major consequences (for our purposes) of disabling STP:

1. You cannot have any loops in your switch topology, or bad things will happen.
2. VLAN prunning on trunks won't work, causing broadcast traffic to be forwarded across trunks that it does not need to cross. We've added features to snmpit to manually do STP's job in this case, so this problem is taken care of.

You must have STP disabled on all switches that are trunked together! If it is enabled on even one, STP traffic will be seen on all of them.

The switch doesn't trust you to use portfast responsibly. So, it has a 'bpdu-guard' feature that helps guard against loops. Turn this feature off with the command

set spantree portfast bpdu-guard disable

Cisco uses a protocol called 'CDP' to discover other Cisco devices. This sends out small packets every two minutes. You can disable it with

set cdp disable <ports>

Ideally, you should only disable CDP on ports that don't have other Cisco devices attached, but in practice, running with CDP disabled on all ports is fine.

Switch ports will, by default, try to negotiate trunking and channeling. Cisco provides a handy macro to disable both of these and also enable portfast on the ports:

set port host <ports>

Setting MAC address aging time

We have found that some experimenters use applications, kernels, etc. that only receive traffic, not send it. This presents a problem, because it prevents the switch from learning which port the node is on, and thus broadcasting traffic for it to every port in the VLAN. This can be solved by 'priming' - i.e. having the receive-only node send some traffic (like an ARP response) at the beginning of the experiment. However, the default aging time of 300 seconds makes this impractical. So, we have disabled this aging, making learned MACs permanent (until the VLAN is torn down.)

You must do this for each VLAN, with the command

set cam agingtime <vlan> 0

For convenience, we've supplied a file in the 'doc' folder of the emulab source tarball called 'no-cam-aging.cfg' that disables aging on VLANs 2-999 (those potentially used by our software). Transfer this file to the switch with the 'copy tftp config' command.

We also suggest that you do this on your control network as well. Part of the booting process leaved the nodes sitting dormant at a boot loader for extended periods of time, so the switch will tend to forget their MACs. Turning off aging is not critical, but we suggest it because it will reduce stray traffic while the switch relearns MAC addresses.

IGMP snooping on the control network

In order for multicast to work correctly, you need to make sure that IGMP snooping is enabled on the control switch. This is needed for frisbee, our disk-loading system. It's up to you whether you want to enable this on the experimental switches. In general, we recommend it so that your experimenters can use multicast, but it does seem that unexpected or malformed multicast packets have an easier time DOSing the switch control processor than unicast traffic. On CatOS, the command is:

set igmp enable

or on IOS:

ip igmp snooping

Cisco has a good reference for IGMP snooping on Ciscos.

Setting up multicast between multiple switches

If you have more than one switch on the experimental or control networks, you may need to do a little setup to get multicast between them. The symptom of this problem is that multicast doesn't work between two nodes on different switches, and if you run 'show multicast groups' on each switch, some will show the group as existing, and others will not.

Run the following command for both sides (i.e. on both switches) of every trunk link:

set multicast router 1/1

(This assums that port 1/1 is your trunk link.) If you are using EtherChannel to bond multiple links together to form a single trunk, you only need to run this command for the first port in the channel.

We had some problems running this command on the trunk on one of our switches. It failed with the error

Failed to add port 2/1 to multicast router port list.

What we finally did to resolve this was to tear down the trunk link and EtherChannel that port was a part of, run the command on it (which succeeded this time), and then build the EtherChannel and trunk back up.

Setting the clock

Since boss is an NTP server, you should set your switches to sync time with it. On CatOS, this is accomplished with

set ntp server 10.11.12.1
set ntp timezone MST -7
set ntp summertime MDT
set ntp summertime enable
set ntp summertime recurring
set ntp client enable
show time

Of course, you'll need to replace 10.11.12.1 with the IP address your boss node uses to talk to the switches (usually its control-hardware interface), and 'MST', -7, and 'MDT' with the names of your timezone and its offset from GMT. If you don't use daylight savings time, leave out the 'summertime' steps, and instead do:

set ntp summertime enable

Watch the output of 'show time' for a while to make sure the clock syncs up. It may take a few minutes.

Setting the login password and the enable (superuser) password.

set password
set enablepass

IOS commands

If you are running IOS on your switches, here are a few notes that may help you 'translate' the above commands. Note that to save changes (stick across reboots) you must do this:

copy running-config startup-config

Interfaces in CatOS are named as module/port, while interfaces in IOS are named as TypeModule/Port. For example, if module 1 has gigabit interfaces, what you would call 1/1 in CatOS is Gi1/1 in IOS. 100Mbit ethernet is 'Fa'. (Really, these are 'GigabitEthernet' and 'FastEthernet' respectively, but you can abbreviate them.)

In order to operate on many interfaces at once, you can issue configuration commands like this:

range gi1/1 - 48, gi2/1 - 48, gi3/1 - 48

which would configure all 48 gigabit interfaces on modules 1, 2, and 3.

The equivalent of 'set port host' (which sets portfast, disables BPDU guard, etc.) is:

switchport host

applied to an interface or a range of interfaces, as in:

interface range gi1/1 - 48, gi2/1 - 48, gi3/1 - 48
switchport host
exit

In order to disable the Spanning Tree Protocol, you would use:

no spanning-tree vlan 1-1005

In order to create a VLAN and set its name:

vlan 10
name control-hardware
exit

In order to set the IP address of the interface in VLAN 10:

interface vlan 10
ip address 10.11.13.183 255.255.255.0
exit

In order to enable an interface:

interface vlan10
no shutdown
exit

In order to remove a VLAN:

no vlan 1000

To put an interface into a VLAN:

interface gi0/1
switchport access vlan 10
exit

To turn on trunking for an interface:

interface gi0/1
switchport mode trunk
exit

To turn off trunking for an interface:

interface gi0/1
switchport mode access
exit

To put interfaces into an EtherChannel:

interface range gi1/41 - 48
channel-group 1 mode on
exit

(Notes: If you want to make more than one channel, give each set of ports a different channel number. And, now, you will configure the whole channel as 'interface port-channel 1'.)

To set the native VLAN on a trunk:

interface gi0/1
switchport trunk native vlan 1

To set the read-write SNMP community string to 'public':

snmp-server community public rw

To globally disable the Cisco Router Discovery (CDP) protocol:

no cdp run

To enable NTP on the switch:

configure terminal
ntp server 10.11.12.1
clock timezone MST -7
clock summer-time MDT recurring
exit
show clock

To set the encrypted password for privileged access.

enable secret password_here

To set the password for privileged access (used when there is no enable secret and when using older software).

enable password password_here

Set password for console access

(config)#line console 0
(config-line)#login 
(config-line)#password password_here

Set password for telnet access (password must be set to access router through telnet)

(config)#line vty 0 4 
(config-line)#login 
(config-line)#password password_here