The Host Route You Can’t Delete: ICMP Redirects + The “clear ip redirects” Command

The Problem

I ran into an issue today where my network management station (NMS) was unable to manage SW1 in the picture above. SW1 was up and running; I could SSH to the SW1 CLI from hosts other than the NMS. Communication between SW1 and the NMS was broken, but there were no other issues reported at the site.

Normally, communication between SW1 and the NMS flows via R1 and across the WAN. R1 is SW1′s default-gateway (not default route, as SW1 did not have “ip routing” enabled). To troubleshoot, I logged into SW1 and did a “show ip route”, which revealed a single entry: 10.0.2.100 (the NMS) would be reached by the firewall, 10.0.1.254. Ah ha: SW1 sending his traffic to FW1 was the cause of the NMS communications failure. FW1 should only have been used in case of emergency to bring up a VPN tunnel, in case the WAN went down. If the WAN was up, traffic sent via FW1 would die due to an asymmetrical route.

The Cause

The source of this host route discovered in SW1 was a puzzlement. How did the switch learn this entry? Did the firewall advertise it? Conceptually this was possible, as the firewall had a backup IPSEC tunnel back to the site where 10.0.2.100 lived, but I also knew that the firewall did not have reverse route injection enabled. Even if it did, it would have been advertising host routes. And beyond that, the switch wasn’t running a routing protocol where it would hear such advertisements.

What about a proxy ARP from FW1? FW1 would only have responded to a proxy ARP if an ARP had placed on the wire by the switch. Since SW1 had a default-gateway, this shouldn’t have happened. Even if it had, FW1 doesn’t happen to respond to ARP requests for remote IPSEC networks. It doesn’t even respond to ARP requests for NAT addresses it’s responsible for very well.

So here’s another important detail that puts all the pieces together. The R1 WAN link went down overnight briefly, and so all of the OSPF routes R1 knew disappeared while the circuit was dead. SW1 syslogs and traps to the NMS at 10.0.2.100. So while the WAN link was down, SW1 originated some traffic for the NMS. This traffic arrived at R1, since SW1′s default gateway is R1. R1, with a dead WAN circuit, no longer had a route covering 10.0.2.100. R1 did have a static default route of his own, however, to 10.0.1.254 – FW1. R1, rather than hairpin route all SW1′s traffic over to FW1 (inefficient), sent SW1 an ICMP redirect message, stating in effect that “the best place to forward traffic for 10.0.2.100 is 10.0.1.254″.

The Solution

The solution might seem obvious. Just clear the route on SW1. And indeed, that is the answer, but surprisingly “clear ip route 10.0.2.100” didn’t work. Neither did “clear ip route *“. If the switch hadn’t been an ocean away and supporting an office full of people, I might have tried a few other riskier things (like enabling “ip routing“), to see what happened. Since avoiding risk was important to me, I dug around on cisco.com, and discovered the “ip redirects” command. I also found the “show ip redirects” command. And where there’s a “show“, there’s quite often a corollary “clear“…and so it was that “clear ip redirects” cleared out that pesky bogus route, and bidirectional communication between SW1 and the NMS was restored.

Preventing Recurrence

WAN circuits do break periodically, so how could I prevent this ICMP redirect message from breaking SW1-to-NMS communication in the future? I could disable ICMP redirects on R1 via “no ip redirects“. One would need to understanding one’s network topology well to know if this was a good idea or not, although generally speaking, it’s a security best practice to disable ICMP redirects as a DoS mitigation technique. I suppose I could filter ICMP redirect messages inbound to SW1 from R1 using an ACL with an ACE like “deny icmp host 10.0.1.1 host 10.0.1.10 redirect“. Note that fiddling with “ip icmp redirects” (hosts vs. subnets) might be appropriate in general, but wouldn’t have prevented the problem in this case.

More ways to prevent this in the future? Perhaps…I’m out of time at the moment to think of more, but chime in with comments.

More Reading

cisco.com: Cisco Guide To Harden IOS Devices

cisco.com: When Are ICMP Redirects Sent?

Ethan Banks
Ethan Banks, CCIE #20655, has been managing networks for higher ed, government, financials and high tech since 1995. Ethan co-hosts the Packet Pushers Podcast, which has seen over 2M downloads and reaches over 10K listeners. With whatever time is left, Ethan writes for fun & profit, studies for certifications, and enjoys science fiction. @ecbanks
Ethan Banks
Ethan Banks
  • http://astorinonetworks.com Joe Astorino

    Great article Ethan. I have one question for you

    “in case the WAN went down. If the WAN was up, traffic sent via FW1 would die due to an asymmetrical route.”

    Why? If the packet went through the firewall over a VPN tunnel to corporate and the return traffic came back to the site over the WAN it would certainly not be ideal and it would indeed be asymmetric routing but it might work. Is there some other ACL filtering or security policy that would prevent that?

    I thought of one I have seen — TCP sequence # randomization on the ASA. So if the initial packets go through the ASA, the ASA will randomize the sequence numbers. When the return traffic comes back over the WAN the NMS thinks it is replying to the doctored TCP sequence but the switch doesn’t know about that sequence number so it resets the connection.

    Just curious!

    • http://packetattack.org Ethan Banks

      State issues will also cause the firewall to drop packets in the asymmetric scenario. SYN comes in across the WAN, SYN/ACK goes out through the firewall, but firewall didn’t see the SYN. So he assumes the SYN/ACK is out of state and drops it accordingly.

  • http://mellowd.co.uk/ccie/ Darren

    We simply disable redirects by default. We want traffic to go the way we want it to, not some other funny way.

    Also as I work for an ISP, we want to ensure customers traffic is actually going the way we intended it to, and not be redirected to some other internal router. If that happens as the other router is advertising reachability to something it doesn’t, you know who will be called to fix it.

  • http://fragmentationneeded.net chrismarget

    Hey Ethan,

    I think your issue is fundamentally a design problem, not a configuration problem.

    You’ve got a non-router device on a network with two candidate gateways with differing capabilities. That non-router has no good way to make a decision about which gateway to use. This can’t be good. If there’s more than one router on a segment (FHRP mechanisms don’t count), hosts shouldn’t be welcome there.

    The fix is to turn the R1-S1 link into a trunk. Use a /30 (or /31!) for your R1-FW segment and put all dumb clients (S1) onto some other subnet/subinterface.

    Also, any suggestion that disabling sending of redirects enhances security is bogus (sorry Cisco!) There was a long thread about this on nanog last summer: http://bit.ly/iRIenF

    The CPU/DOS argument doesn’t make sense: why disable a feature that can only be hammered from a connected interface while leaving on ICMP responses to things from off net (ping, too big, no route, etc…)?

    The only good reason to disable sending of redirects is if you’re using BFD. Disabling redirects to buy back control plane horsepower suggests that there are deeper issues :-)

    • http://packetattack.org Ethan Banks

      The “design,” as such, isn’t one – so “fundamentally” ;-) I agree. Where I ran into this issue, there was no design. Whoever put this together just plugged things in and hoped. When they got it to “work”, they walked away. Time passes, they quit, I get hired, and I’m cleaning up site by site, this specific site not one I’ve gotten to as yet. They need new equipment all around, as well as conformity to the standards I’ve put in the place at the larger sites, some of the equipment being incapable of certain configuration until it’s replaced. Point being, this article wasn’t about design or lack thereof. It was about how an ICMP redirect message populates the routing table of a switch, and how to handle it. Just one of those odd little things I ran into working on a less than ideal situation.

      As far as the NANOG thread goes, Cisco’s point seems valid to me, at least in the context of the vacuum where they make the recommendation. “A malicious user can exploit the ability of the router to send ICMP redirects by continually sending packets to the router, forcing the router to respond with ICMP redirect messages, resulting in an adverse impact on the CPU and performance of the router.” Can that happen as Cisco describes it? Sure. Would you still want to disable redirects? Does it make sense in the larger scheme of things to disable redirects? As always, it depends.

      Cisco’s hardening guide talks about a series of security challenges, often DoS related, and how to mitigate them. As such, it’s a useful document. It makes one aware of what could happen in certain circumstances and strategies worth considering to help maintain control of your network infrastructure should it happen. If your infrastructure is being attacked, you might need to know how to get the control plane back, or better yet, how to prevent CPU overload in the first place.

      One of the areas where I feel the hardening document is weak regards CoPP – they mention it in passing, illustrate one of the least powerful features, but that’s about it (yes, there’s a link to the deployment guide). “Control Plane Policing can be used in order to identify the type and rate of traffic that reaches the control plane of the Cisco IOS device. Control plane policing can be performed through the use of granular classification ACLs, logging, and the use of the show policy-map control-plane command.” CoPP is a bit of a challenge to implement as there’s no “one size fits all” template, which I assume is why they don’t get into it in more detail in that specific document (although they do elsewhere). But – I think a good CoPP policy can move into the realm of actually managing control plane traffic as opposed to just disabling features because a document said to. I wish Cisco would push CoPP a bit more.

  • http://fragmentationneeded.net chrismarget

    Hey Ethan,

    I didn’t mean to suggest that you were responsible for the mess, only that configuring around the problem didn’t seem like the way to handle it. Hosts just don’t belong on transit links.

    I’ll confess that it surprises me that redirects are so, well, permanent.

    Folks in the nanog thread went to great lengths to distinguish between sending redirects, and acting on them. As far as I’m aware, Cisco IOS doesn’t offer a way to ignore incoming redirects in this situation. Disappointing, huh?

    While I certainly can’t argue with the idea that not sending redirects saves the processing associated with sending them, protecting against this sort of attack just doesn’t seem particularly meaningful/useful considering how many nearly identical attack vectors there are, and that a first-hop router probably shouldn’t think that some other device on an access LAN is the best path to anywhere.

  • Klaas

    Great post! I ran into this exact issue and could not get the host routes off the interface I was on without resetting the interface and losing connection to the router. Your command helped me.

    • kk

      Very good info. I ran into this problem on some pure L2 switches (2960G with 15.0.2) and even resetting the interface did not resolve the issue.

  • SMiller

    I’m running into this issue as well. I have an MPLS network with a managed router at the site. We are running the EIGRP routing protocol. We also have a local internet connection because we don’t want to backhaul internet traffic.
    When the MPLS link bounces we get IP Redirects on several different chatty devices trying to talk back to the data center through the MPLS link.
    When the MPLS link restores its connection the devices that received the IP redirects are stuck with those specific routes in the IP redirect table.
    I used your suggestion to remove the IP redirects and all was good. I’m having my MPLS vendor disable ICMP redirects from all the managed MPLS routers at our remote sites.
    You would think that there would be a default timer set on these devices to drop those IP redirects from the table. Unfortunatly we have ShoreTel devices as well as a few Cisco AP’s that wouldn’t drop those entries. The only thing that seems to be a long term fix is to remove the ICMP redirects from the router.
    Thanks for the good information Ethan.