Ideally, all of your Cisco router configurations run like clockwork and you rarely need to troubleshoot them. That's the way it should be, since these devices usually work as the pillars of network connectivity within an organization. But if your Cisco routers do have problems, you need to be able to come to a resolution quickly and effectively. That's why we're going to look at how to use the Cisco IOS debug commands.
This article is intended for the intermediate level network administrator who has never encountered circumstances that would require debugging the Cisco IOS. But if you've used debug before, perhaps this article will give you some suggestions for ways to use debug that you haven't previously tried.
Key points
Before I introduce 10 ways to use Cisco IOS debug, here are a few tips and caveats:
- · The techniques I'm including are not the only ways to use debug. In fact, there are hundreds of ways to use debug. List A shows all the possible subcommands that can be debugged in IOS 12.2 (the version I was running). You will see that there are more than you would ever want to use. In addition, each subcommand has its own subcommands. I'll be focusing on some of the debug commands I consider most common and useful.
- · Using Cisco IOS debug can be dangerous and should first be used on a test router and then used carefully on a production router. I don’t want anyone to get fired because he crashed the company’s production router after trying debug commands from this article.
- · By default, debug messages will go to the console and will not be sent to any log. To log debug output (or any system output) to the IOS system log, you must turn on logging buffered in global configuration mode. You can see if logging is on by issuing the show logging command on your router. The output looks like List B. As you can see, there are three kinds of logging: console, monitor, and buffer. There are also different levels of logging. You can read more about this here.
- · If you want to see debug output on a terminal (or Telnet session) that is not the console, use the terminal monitor command.
- · You can also set up “conditional debugging” to debug only when certain conditions apply. See Cisco's Troubleshooting and Fault Management to learn more about this.
- · If you turn on some debugging that is filling up your screen, the quickest way to turn it off is by running the un all command. The un is a built-in macro for undebug.
Now let’s move on and look at 10 ways to use Cisco IOS debug. We'll start with number 10 and work our way up to the most useful debug command.
10. Debug all
By far, the most dangerous debug command is debug all. This command will most likely crash a production router. The Cisco IOS is nice enough to ask, “Are you sure?” before doing it. Only use this command if you are desperate, you know you are on a test router, and/or you are having trouble isolating the proper subcommand to debug. To satisfy your curiosity of what it looks like to run it, List C shows the output from executing the command and then issuing show debug to see what it turned on.
9. Debug with ISDN and dialer
Debugging is helpful when troubleshooting Cisco router dialup configurations. In my experience, these technologies rarely seem to work the first time. The best way to configure IOS dialup technologies (or any networking technologies) is layer by layer, testing each layer after configuring it. I try to think of a multilayer cake, where the icing is the successful testing of each OSI layer after each configuration. When something doesn’t work, debug is usually a must. Useful debug commands for ISDN and dialup networking appear below:
- · debug isdn events debugs all ISDN events. Start with this one first before trying q921 or q931, as those are very verbose and events may tell you all you need to know.
- · debug isdn q921 is Layer 2.
- · debug isdn q931 is Layer 3.
- · debug dialer events will tell you about the calls (i.e., the reason the call was initiated and/or disconnected). Here is some sample output:
00:18:52: BRI0/0 DDR: Attempting to dial 8358661
00:19:22: BRI0/0:2 DDR: disconnecting call
This can be especially helpful if you have a router that keeps calling when you don't want it to.
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:39:24: BRI0/0 DDR: ip (s=1.1.1.2, d=1.1.1.1), 100 bytes, outgoing interesting (ip PERMIT)
00:41:09: BRI0/0 DDR: cdp, 273 bytes, outgoing uninteresting (no list matched)
8. PPP authentication
If you configured PPP authentication on your dialup lines (which you should, for security reasons), you could end up with username/password mismatches on each side. Finding out that this is the reason that the line won’t come up can be impossible without using debug ppp authentication.
Here's some sample output of missing username/password information on one router:
00:32:30: BR0/0:1 CHAP: O CHALLENGE id 13 len 23 from "r2"
00:32:31: BR0/0:1 CHAP: I CHALLENGE id 2 len 23 from "r1"
00:32:31: BR0/0:1 CHAP: O RESPONSE id 2 len 23 from "r2"
00:32:31: BR0/0:1 CHAP: I FAILURE id 2 len 26 msg is "Authentication failure"
This is sample output of an incorrect password on one router:
00:47:05: BR0/0:1 CHAP: O CHALLENGE id 25 len 23 from "r2"
00:47:05: BR0/0:1 CHAP: I CHALLENGE id 19 len 23 from "r1"
00:47:05: BR0/0:1 CHAP: O RESPONSE id 19 len 23 from "r2"
00:47:05: BR0/0:1 CHAP: I FAILURE id 19 len 25 msg is "MD/DES compare failed"
7. Debug {topology} packet
As I mentioned above, it is best to troubleshoot networking technologies using a layered approach, working your way up the OSI model from Layer 1. (I give credit to Bruce Caslow and his book Cisco Certification: Bridges, Routers, and Switches for CCIEs for carefully documenting this approach.)
Using that model, for whatever network topology you are working with, you can use the debug command to see that your Layer 2 packets are being encapsulated properly. (Of course, I'm assuming that all the cables are connected at Layer 1.) For instance, let's say that you work with frame-relay and you're having trouble getting packets across a link. You've verified that the link is up, but there is still no data passing on it. You could try
debug frame-relay packet
Then, you could attempt to ping the interface on the remote router. If the ping passes, you will get the following:
01:03:22: Serial0/0:Encaps failed--no map entry link 7(IP)
This tells you that encapsulation failed for the IP packet into the frame-relay protocol. It even tells you that it failed because there is no frame-relay map statement. If you fix that and try it again, you might find that you get no frame-relay errors. But then the packet still may not pass. Thus, you'll need to go up to Layer 3 and try:
debug ip packet
You might get this output:
01:06:46: IP: s=1.1.1.2 (local), d=11.11.11.11, len 100, unroutable
This tells you that there is no IP route for your packet to traverse. Thus, you know you need to add a route.
You can insert whatever networking topology you are working with on the debug {topology} line. Some other possibilities are:
- · debug atm packet
- · debug serial packet
- · debug ppp packet
- · debug dialer packet
- · debug fastethernet packet
6. Debug crypto (IPSec and VPN features)
I won’t go into a lot of detail and examples here since IPSec and VPN are huge topics that can get very complex. But I do want to point out that you can troubleshoot IPSec connections with debugging commands. Because of the popularity of VPN and IPSec, it is important to know how to use debug commands for these technologies. Some of the possibilities are:
- · debug crypto isakmp
- · debug crypto ipsec
- · debug crypto engine
- · debug ip security
- · debug tunnel
In addition, debug ip packet, as mentioned below, is useful in troubleshooting IPsec connections.
If you want to learn more on this topic, I recommend going to Cisco’s IP Security Protocols Section, choosing the protocol you are interested in (IPSec, IKE, etc.), and looking at the different configurations along with their recommended troubleshooting methods.
5. Debug IP routing
Using debug with IP routing protocols can be useful as well. For example, to know if you have a route flapping (a route being added and removed frequently), you can use debug ip routing.
If you had a flapping route, you might see something like:
01:30:56: RT: add 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:31:13: RT: del 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:31:13: RT: delete subnet route to 111.111.111.111/32
01:31:13: RT: delete network route to 111.0.0.0
01:32:56: RT: add 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:33:13: RT: del 111.111.111.111/32 via 12.12.12.11, ospf metric [110/65]
01:33:13: RT: delete subnet route to 111.111.111.111/32
01:33:13: RT: delete network route to 111.0.0.0
This could indicate a routing loop in your network. On the other hand, it could indicate a link going up and down, perhaps a bouncing dialup interface or frame-relay interface.
4. Debug ip {routing protocol}
You can insert many options with debug ip. Issuing the debug ip ? command will display them. We've included the options in List D.
Most routing protocols (e.g., OSPF, EIGRP, IGRP, and BGP) are included in this list. Each has its own extensive options that can be debugged. (You can see all the options for a particular protocol by issuing the debug ip {routing protocol} ? command.) For instance, the only way to know that your two OSPF routers are not forming an adjacency due to mismatched authentication types is to issue debug ip ospf adjacency.
You might see the following output, telling you that the authentication types are mismatched:
01:39:46: OSPF: Rcv pkt from 12.12.12.11, Serial0/0 : Mismatch Authentication type. Input packet specified type 0, we use type 2
In this case, if it weren't for debug, you could be scratching your head for a while.
3. Debug list
An interesting debug command that isn’t widely known is debug list. This command doesn’t actually debug anything. Instead, it sets an interface and/or access list for the next debug you enter. So if you are going to issue some command that you know will produce a lot of output but that doesn’t have an option to limit that output with an access command, you can first issue the debug list XXX command (where XXX is an access list number you have already created) and then run the command you want to debug on. An example would be:
debug list 1
debug dhcp detail
DHCP client activity debugging is on for access list: 1 (detailed)
2. Log an access list to system or syslog
You can use the log option at the end of an access list to log packets that are permitted or denied. This could enable you to log packets that are denied on a router being used as a firewall or to control access at an important network point. Remember that a firewall doesn’t have to just protect you from the Internet; it can protect higher security departments (such as accounting), be placed between two business partners, or just keep some traffic (like P2P traffic) off your network. While this isn’t technically a debug command, it can certainly be used for debugging and troubleshooting. Let's look at a couple of examples.
First, suppose that you want to allow only BGP traffic in your network and keep track of all other traffic that attempts to enter a link. Here's a sample configuration:
Interface Serial 0/0
Access-group 100 in
access-list 100 remark Begin -- Allow BGP IN and OUT
access-list 100 permit tcp host 1.1.1.1 host 2.2.2.2 eq bgp
access-list 100 permit udp any host 2.2.2.2 gt 33000
access-list 100 remark End
access-list 100 deny ip any any log
All traffic will go to your Cisco router’s log if you have “logging buffered” turned on or to a syslog server if you configured one.
For the second example, let's say you are trying to nail down an access list to permit only the proper traffic to go over your dialup link. You use an access list and log the existing traffic to get an idea of what is being carried so that you can configure the correct access list. Here's a sample configuration:
Interface BRI 0/0
Access-group 100 in
Access-group 100 out
access-list 100 permit ip any any log
If you look at the output in the router’s log, you will see that an ICMP packet and a TCP packet went across your link:
02:03:43: %SEC-6-IPACCESSLOGDP: list 100 permitted icmp 1.1.1.2 -> 1.1.1.1 (0/0), 1 packet
02:06:25: %SEC-6-IPACCESSLOGP: list 100 permitted tcp 1.1.1.2(0) -> 1.1.1.1(0), 1 packet
1. Debug IP packet detail XXX (access list number)
My number-one debug technique is using an access list to see all the traffic going to and from some destination, just like a basic sniffer/protocol analyzer. The limitation to this one is that you can’t see the packet payload and can see almost nothing of the headers.
Since you can do this with an access list, you can nail it down to a particular host, protocol, port, or network range, as well as using a uni- or bi-directional access list. I know it isn’t a real protocol analyzer, but it could still come in handy—and it's already built into the IOS. The sample configuration below shows the logging of all Telnet packets to your router. (These could just as well be a host on one of your router’s networks.)
access-list 101 permit tcp any any eq telnet
debug ip packet detail 101
IP packet debugging is on (detailed) for access list 101
List E shows the sample output of what the debug output would look like.
You can see from the output that you get the IP source, destination, interfaces, source and destination port numbers, sequence numbers, acknowledgement numbers, window sizes, and TCP communication flags (SYN, ACK, FIN).
Summary
I have covered 10 of the many uses of Cisco IOS debug, which I hope will help you in future troubleshooting. Cisco IOS debug isn’t just for Cisco TAC engineers; all Cisco administrators can use it to troubleshoot just about any configuration that's put into a Cisco router.
No comments:
Post a Comment