Ciscovoodoo
Some notes on how to diagnose problems on Cisco switches running CatOS
Looking at switch stats/status for a node interface.
Mac may have a wrapper for this, but here is the guts of what you have to do. To look at the switch port for a particular enet card, first figure out what switch port it is connected to!
mysql tbdb select * from wires where node_id1='tbpcXX';
What you care about are node_id2/card2/port2. Node_id2 tells you which cisco: "cisco" is the testbed cisco (tip test) and "cisco2" is the control net cisco (tip control). card2/port2 give you the info you need for the cisco-style card/port name (e.g., 3/39).
[Note:
There are also a couple of other easier ways to do this:
Rob has created a command called if2port:
108 paper:~> if2port
Usage: /usr/testbed/sbin/if2port <node | node:if | node if>
109 paper:~> if2port tbpc06
+--------+-----+--------+-----+-----+
|node_id1|card1|node_id2|card2|port2|
+--------+-----+--------+-----+-----+
|tbpc06 | 0|cisco | 3| 42|
|tbpc06 | 1|cisco | 3| 44|
|tbpc06 | 2|cisco | 3| 46|
|tbpc06 | 3|cisco | 3| 48|
|tbpc06 | 4|cisco2 | 3| 11|
+--------+-----+--------+-----+-----+
5 rows processed
Another useful way is with 'snmpit -l -debug'. This will give all the
tbpcXX:Y <==> ciscoport translations for any port currently in use,
as well as listing the VLANs currently set up:
111 paper:~> snmpit -l -debug
DEBUG MODE ON: Set to level 1
Command line was: snmpit -l -debug
Use of uninitialized value at /usr/testbed/bin/snmpit line 1281.
READING TRANSLATIONS
Opening SNMP session to 155.101.128.175...Succeeded
Getting VLAN info...
Got default vtpVlanName 1.1 (1) default
Got tact-reserve2-l0-0 vtpVlanName 1.2 (2) tact-reserve2-l0-0
Got brandeis-BuddyCache-l0-0 vtpVlanName 1.3 (3) brandeis-BuddyCache-l0-0
Got tact-reserve2-l0-1 vtpVlanName 1.4 (4) tact-reserve2-l0-1
...
Got 3 vlanPortVlan 6.34 3 ('6.34' == tbpc23:0)
Got 19 vlanPortVlan 6.41 19 ('6.41' == tbpc21:0)
Got 19 vlanPortVlan 6.42 19 ('6.42' == tbpc24:0)
Got 19 vlanPortVlan 7.25 19 ('7.25' == tbpc25:0)
Got 19 vlanPortVlan 7.34 19 ('7.34' == tbpc29:0)
Got 3 vlanPortVlan 7.41 3 ('7.41' == tbpc27:0)
Got 19 vlanPortVlan 7.42 19 ('7.42' == tbpc30:0)
Got 19 vlanPortVlan 8.33 19 ('8.33' == tbpc32:0)
Got 5 vlanPortVlan 8.42 5 ('8.42' == tbpc36:0)
Got 8 vlanPortVlan 9.33 8 ('9.33' == tbpc38:0)
ID Name Members of VLAN
--------------------------------------------------
1 default
2 tact-reserve2-l0-0 tbpc04:0 tbpc10:0
3 brandeis-BuddyCache-l0-0 tbpc20:0 tbpc22:1 tbpc23:0 tbpc27:0
4 tact-reserve2-l0-1 tbpc04:1 tbpc13:0
5 agile-test001-l0 tbpc36:0
6 janos-moab-l0 tbpc02:0 tbpc03:1
7 janos-moab-l1 tbpc01:0 tbpc03:0
8 agile-afreenet-l0 tbpc06:0 tbpc38:0
12 magi-test1-l0 tbpc07:0 tbpc11:0
18 _mylan tbpc15:0 tbpc16:0 tbpc18:0 tbpc19:0
19 __mylan tbpc21:0 tbpc24:0 tbpc25:0 tbpc29:0 tbpc30:0 tbpc32:0
The interesting translations are just above the vlan table, in the
far right hand columns. For example, the last line before the table
indicates that in vlan 8 is port 9.33, which belongs to tbpc38:0.
]
Armed with this info, tip to the correct cisco, login and enable. Then you can do:
show port status card/port
or
show port card/port
for everything. If, say, the port is disabled, you can do:
set port enable card/port
Checking on the firewall rules
You have to login to the "control" Cisco and then "session 15" to connect to the Router module.
While at the Router> prompt, you will get any "access denied" type messages that the router produces, ala:
23w2d: %SEC-6-IPACCESSLOGP: list control-shark denied \
udp 0.0.0.0(0) -> 255.255.255.255(0), 602 packets
If you suspect that some rule is preventing your traffic from getting through, then try generating your traffic while you are connected to the router and see if you get errors.
To see the whole lists in all their ugliness, type:
show ip access-lists
The rules are pretty straightforward. First match wins. Netmasks are bass-ackwards (intead of 255.255.255.0, you would use 0.0.0.255). Each list is applied both on entrance and exit to the like-named control network VLAN.
Finding MAC address information (CatOS)
To find which port a given MAC address is on type (on the switch console):
show cam <MAC>
where MAC is colon-seperated, like 08:00:2b:81:62:d3.
To show all MAC addresses in a given VLAN, type:
show cam dynamic <VLAN>
where VLAN is the number, not the name.
Finding MAC address information (IOS)
To find which port a given MAC address is on type (on the switch console):
show mac-address-table address <MAC>
where MAC is dot-separted, like 0011.bc81.e400
To show all MAC addresses in a given VLAN, type:
show mac-address-table vlan <VLAN>
where VLAN is the number, not the name.
Deleting a "sticky" ARP entry
If you should ever be so unfortunate as to have to replace a faulty shark, in addition to recording the new MAC address in the DB and DHCP config file, you may also need to clear it from the router module. If you fire up a new shark, and it says that it cannot get its DHCP info, this is likely the problem. To find out, login to the control Cisco and "session 15" to get to the router module. You should start seeing periodic
24w2d: %IP-3-STCKYARPOVR: Attempt to overwrite Sticky ARP entry: \
155.101.130.73, hw: 0800.2b81.62d3 by hw: 0800.2b81.611b
messages. To clear the arp entry (actually the whole cache), enable at the Router> prompt and then do "clear arp".
Replacing a node/NIC
If you replace a node, you'll need to change the secure MAC address for that port. The following command should work:
Console> (enable) set port security 3/1 enable 01-02-03-04-05-06
Of course, use the real port number and MAC address (noting the funky MAC syntax). Note that you will proably also need to use the 'Sticky ARP Entry' clearing procedure covered above.
Checking on port security
To find out what MAC address(es) are associated with a given port. use:
Console> (enable) show port security <port>
To find out if a given port has been disabled for being a Bad Boy (tm) w/ respect to MAC addresses, use:
Console> (enable) show port <port>
- the state will be 'disabled', and you should see some information on the security violation
To re-enable a port after it has been disabled due to security violations:
Console> (enable) set port enable <port>
To disable security for a port:
Console> (enable) set port security <port> disable
Manual VLAN configuration - From the switch command line
To see a list of all configured VLANs, use:
Console> (enable) show vlan
On the control net, all of the VLAN names should be self-explanatory
Adding a port to a VLAN is very easy. Just type:
Console> (enable) set vlan <NUM> <PORT>
... where <NUM> is the number of the VLAN, and <PORT> is the port (you can use the 'if2port' script to get the port number)
To 'remove' a port from a VLAN, set it to VLAN 1.
To create a new VLAN, use:
Console> (enable) set vlan <NUM> name <NAME>
... where <NUM> is some unused VLAN number (use 'show vlan' to find one), and <NAME> is some descriptive string
To delte a VLAN use:
Console> (enable) clear vlan <NUM>
... where <NUM> is the VLAN number (duh!) NOTE: This puts all of the VLANs ports back into VLAN 1, and disables them. Use 'set port enable <PORT>' to re-enable it
Cloning all traffic from a port or VLAN to another port
Pick a port to recieve the traffic - let's call it <monitor>
To forward the traffc from one port:
Console> (enable) set span <port> <monitor>
To forward the traffic from an entire VLAN:
Console> (enable) set span <vlan #> <monitor>
NOTE: You might want to append 'rx' or 'tx' to the VLAN command line, or you'll get doubles of everything (incoming and outgoing both). <rx> means into the switch on <port>, <tx> means out of the switch.
NOTE2: If you want to be able to send traffic from the monitor port, you'll need to append 'inpkts enable' ala:
Console> (enable) set span 18 3/9 rx inpkts enable
To stop cloning:
Console> (enable) set span disable <monitor>
NOTE: as of 10/1/09 the span places are:
cisco2: 2/17 to fxp0 on nslow.flux.utah.edu (100Mbs)
fluxx: fa0/12 to em1 on axl.flux.utah.edu (100Mbs)
NOTE: as of 10/11/07 bge0 on oboss has been taken over and oboss is now nslow.flux.utah.edu. You can use 2/17, which is 100Mb fxp0 on nslow.
NOTE: as of 12/14/06 we have bge0 on oboss attached to the control net switch (cisco2) so that it can be used for spanning. Its cisco2 port is 9/13, use that for <monitor>. Also note that you will have to have the interface configured in order to run tcpdump on it. I just do:
ifconfig bge0
no IP info is necessary. Then:
ifconfig bge0 down
when I am done.
Span on IOS
config mointor session 1 dest int gi8/10 mointor session 1 source int gi3/3
If you need to change anything, you should first do:
no mointor session 1 source int gi3/3
Multiple span sessions
You can setup multiple span sessions on a switch. Just add 'create' to the end of your setup line:
Console> (enable) set span <port> <monitor> create
To tear down a specific span session, name it by the <monitor> port as above:
Console> (enable) set span disable <monitor>
Span on trunk ports
You can also span traffic on a trunk port or even a set of bonded trunk ports. For example, on our experimental switches we have four bonded Gb ports, 2/1-4, as an inter switch link, so you could see all traffic coming in from the trunk with:
Console> (enable) set span 2/1-4 5/5 rx create
You can even pick out a specific VLAN or VLANs from the trunk. So if you only care about VLAN 300 and 400:
Console> (enable) set span 2/1-4 5/5 rx filter 300,400 create
I haven't tried this, but according to the document cited below (#12), you can preserve the VLAN tagging when spanning a trunk by putting the monitor port into trunk mode first.
Good reference for span:
http://www.netplusinc.com/misc/CiscoSPAN.pdf
How to setup span for IOS on the 6500 series:
How to setup span on the 2950 switches: http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015c612.shtml#topic5
Stuck 1000/100/10 ports [ ISI - faber@isi.edu ]
Occasionally Gigabit switch ports seem to get stuck in a state where they do not output packets to the host connected to them. We're not sure why this happens (we're looking), but there is a fix. Simply toggle the speed using
set port speed module/port 1000
and then back to whatever speed it was originally set for. Of course if it was originally 1000 Mb/s the first set should be to 100 and then back to 1000. We have seen ports lock up this way configured to both 1000 and 100 Mb/sec.
You may have to both toggle the switch speed and re-ifconfig the host NICs to re-establish connectivity.
Good reference for making nodes boot quickly
(ensuring that port spanning, trunking, etc. are off on the switch port)
http://www.cisco.com/warp/public/473/12.html
The gist is that we should use the convenient:
set port host <port ...>
To turn on fastport (disable spanning tree) and turn off channeling (combining multiple ports to make a fast link) and trunking (a single port serving multiple VLANs).
Good reference for how IGMP snooping (used for multicast) works
http://www.cisco.com/warp/public/473/22.html
The most interesting bits are near the bottom in the part with the heading "IGMP Snooping", and some more in "Practical Example of IGMP Snooping"
IOS for guys who know CatOS: (ie. me)
http://www.cisco.com/warp/public/cc/pd/si/casi/ca6000/tech/cat65_wp.htm
For into on how the switches do load balancing on etherchannel (aka "trunk") links:
http://www.cisco.com/warp/public/473/4.html
snmpit 'unable to find channel information' errors
Sometimes, for no reason we can discern, a switch stops returning information about what EtherChannel a port belongs to. When this happens, snmpit is unable to deal with trunks that cross the channel, and will return an error. Note that this failure *only* occurs through SNMP. If you do a 'show port channel' (or IOS equivalent) on the command line, everything looks fine. The simple way to clear these up is to delete and re-create the channel. Luckily, you don't have to touch the configuration of the trunk that runs on it (though this will temporarily disrupt traffic):
set port channel 2/9-12 mode off set port channel 2/9-12 mode on
... of course, use the actual ports that your channel runs on.
Useful commands for debugging high CPU usage on routers (IOS)
- show proc cpu
- shows %ages of CPU used by various processes. If CPU usage is high, and process usage doesn't add up to total usage, then the extra time is in interrupt handlers, where 'fast path' forwarding (and other things) take place
- show ip traffic
- show router-wide stats on traffic, including the reasons for dropping packets. 'Encapsulation failed' (for drops) typically means that the router wasn't able to ARP for the destination
- show ip interface
- will tell you more about the options, such as fast switching, enabled on an interface. You definitely want fast switching enabled, probably want CEF switching if your device supports it, and might want 'flow' or 'netflow' if you are using big ACLs (since it effectively caches firewall decisions)
- show ip cache flow
- shows the contents of the 'newflow' per-flow routing cache, including an interesting histogram of packet sizes. If you want to look at entries for a particular host, you can use '| include <IP>'
Cisco page about troubleshooting high CPU utilization:
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml
If it's from interrupts:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml
More about performance tuning, the various switching paths, etc:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a7306.shtml
Pasting the output of these commands into Cisco's Output Interpreter tool (on the web, registered users only) is very helpful