Cisco ASA 8.3+8.4 Hairpinning NAT Configuration

I ran into an issue over the weekend where a VPN client was unable to access a remote office connected via an L2L tunnel terminated on the same firewall. The symptoms were straightforward enough. The client was unable to either ping or open a URL at a specific server at the remote office, although this connectivity used to work. In this example, VPN client 192.168.100.100 was not able to access server 10.11.12.1, although access to resources in the 10.10.0.0/16 network was fine.

I confirmed the remote office firewall was unlikely to be the issue; the remote firewall had seen no changes. As I knew the headquarters Cisco ASA firewall HAD seen a few changes, that’s where I focused my attention. After reviewing the headquarters firewall rulebase, I knew that the VPN client IP pool had permission to access resources in the remote office.

Monitoring the firewall logs, I spotted several “110003: Routing failed to locate next-hop for protocol from src interface:src IP/src port to dest interface:dest IP/dest port” messages tied directly to the VPN client trying to open a socket to 10.11.12.1. So, I reviewed the firewall routing table with “show route” and “show asp table routing” and found no issues…not that I expected to. If the routing table was having a problem, connectivity issues would have been more widespread.

Of course, NAT sprung to mind as a potential issue, but I couldn’t see an obvious problem. There was a NAT that exempted the entire VPN client pool from being translated to any RFC1918 destinations. As this clearly covered the remote office IP range, I was a little stumped. This confusion was compounded by the fact that the connectivity used to work. A perplexing issue.

After a little googling around, it seemed dead certain that the issue was NAT, although there was no one single obvious solution anywhere that I ran across. XKCD knows where I was.

I felt a little chastened, as I tend to go for the documentation first anyway, although in fairness I did search cisco.com – I just hadn’t found any answers. But with Google being less than helpful, I went back to the Cisco docs again, where with great joy, I found The Answer. Answers are good. Answers make me happy. Like this.

Take a read here. A couple of highlights caught my eye:

  • The “same-security-traffic permit intra-interface” is required. Fair enough, easy to implement, makes sense, and I’d already done that. No problem.
  • Now the documentation got confusing because of two conflicting statements:
  1. “When the ASA sends encrypted VPN traffic back out this same interface NAT is optional. The VPN-to-VPN hairpinning works with or without NAT.” Okay – so I don’t need to write a NAT statement for the hairpinned traffic. NAT is optional, right? But then you next read…
  2. “To exempt the VPN-to-VPN traffic from NAT, add commands that implement NAT exemption for VPN-to-VPN traffic.” Uh, hang on. So I *do* need a NAT statement?

From my experience, I believe that, yes, you need a NAT exemption statement. I think all Cisco is trying to say is that you don’t have to actually translate the source or destination address into something else to be able to get through the hairpin.

Writing a NAT exemption statement is not an unusual thing to have to do in an ASA, but the magic in the context of hairpinning is in defining the ingress and egress interfaces. In a hairpin path, the traffic flows in and out the same interface. While I did have a NAT statement that matched source and destination addresses in question, the interfaces were only suitable for handling source VPN client to destination headquarters network traffic…not traffic headed from VPN client to the remote office network. Therefore, I needed a NAT statement like this: “nat (outside,outside) source static client_vpn_pool client_vpn_pool destination static remote_office_net remote_office_net“.

The order of the NAT statement also mattered, as NAT statements are processed in order. Once I moved my new NAT statement to the top of the list, the issue was resolved.

Now…this all left me with some questions which I’m planning to test in a lab to see verify ASA behavior under 8.3 and 8.4. More posts to come on ASA NAT, and your comments welcome if you have experiences of your own to share on ASA NAT under the new paradigm.

Ethan Banks
Ethan Banks, CCIE #20655, has been managing networks for higher ed, government, financials and high tech since 1995. Ethan co-hosts the Packet Pushers Podcast, which has seen over 2M downloads and reaches over 10K listeners. With whatever time is left, Ethan writes for fun & profit, studies for certifications, and enjoys science fiction. @ecbanks
Ethan Banks
Ethan Banks
  • http://twitter.com/vRobM Rob Marković (Robi)

    I think the doc difference is coming from:
    1. NAT is not required, and
    2. If NAT is in use (already), you need to decide how to deal with it.

    • http://packetpushers.net/author/ecbanks Ethan Banks

      I like that interpretation, too. Will get to the bottom of it.

      • Guest

        This seems to be accurate with my testing of ASA 8.3+. You you have not defined any NAT on either of the interfaces involved in the traffic flow then you do not need to add any NAT statements. However, if like most people you NAT your inside networks to public IPs for internet access then you need to add a Twice NAT statement to the top of the list as illustrated in your post. Good to know about the (outside,outside) part though. In pre 8.3 that wasn’t necessary and I found your post while trying to ponder how the NAT would need to be defined as far as the interface names.

  • http://twitter.com/mrdoro Lukasz Dorosz

    In my scenario I’ve got an ASA in HQ and a few remote L2L connections + VPN Clients.
    I don’t use a NAT. To permit a traffic like in your scenario I only performed a “same-security-traffic permit intra-interface” command and it resolved my issue.

    • http://packetpushers.net/author/ecbanks Ethan Banks

      Okay – that brings back the confusion a bit, but also echoes behavior I’ve seen in the ASA when traffic is traversing other interfaces. I’ll try to replicate your experience in a lab environment. It makes sense to me that if you aren’t actually re-writing an IP address, then you shouldn’t need a NAT statement at all. I’ll hack around on an ASA and see what I can come up with as consistent behavior.

  • Anonymous

    So what changed that caused the client to no longer work? Or did it REALLY ever work? And that cartoon is scarily familiar…

    • http://packetpushers.net/author/ecbanks Ethan Banks

      I know that an object-group referenced in a global-ACL, crypto-ACL, and NAT was updated with a new network. That seems to have been what triggered the NAT failure. Why that would have broken it? Honestly, I’m not sure.

      My co-worker (an extremely competent engineer) told me that it had been tested previously, and I have no reason to disbelieve him. Therefore, although I never personally saw it working previously, I believe it was.

  • Marco Mineto

    Hi Ethan, this article was really helpfull for me. It worked just perfectly.
    Thanks a lot for your contribuition

  • Sebastian S

    I ran into this same issue, in my case it was Client VPN hairpinning to L2L VPN networks. I had the outside->outside nat exemption configured however I was seeing route lookup failed for host x.x.x.x outside to x.x.x. inside. The culprit ended up being that my inside to outside exemption included an any statement. nat (inside,outside) source static any any destination static obj-172.20.2.0 obj-172.20.2.0. Adjusting the exemption to more specific subnets on the inside that did NOT include the RA vpn pool networks resolved the problem. DAMN YOU ANY ANY Y U get me again.

    • Ron Wilkerson

      thanks for sharing this. i had the same issue. i still don’t understand why the (inside,outside) nat affected the (outside,outside) definition.

  • Kevin Frey

    Thank you. Also thanks for illustrating your line of reasoning and tone – always love to hear from like minds.

  • Nubos

    Hello, guys!

    subnet 10.0.0.0 255.0.0.0I have the same scenario -> HQ ASA-5520 (8.4.5ED) as a HUB for ezVPN users and L2L spokes (branches).

    The exemption rule is in the list as a single NAT record.

    Tunnels is up: ez could ping/rdp to HQ, Any from HQ could use branche resources (open rdp/file shares, etc), but traffic from ez to spoke dosen’t go.

    Here’re the rules:

    object network ezvpn (ezvpn pool)
    subnet 10.0.254.0 255.255.255.0

    object network tln (branch)
    subnet 10.5.0.0 255.255.0.0

    nat (outside,outside) source static ezvpn ezvpn destination static tln tln

    same-security-traffic permit intra-interface

    no effect.

    IKE&IPSec sa’s are up and working.

    What could it be?!

  • Jerm

    Great article, just what I needed but in reverse ! I’m upgarding a 5505 ASA with 7.2 code to a new 5512 ASA with 9.1. The issue I’m having is with (inside,inside) hairpinning. Implementation is tonight , keeping my fingers crossed.

  • suhas

    hi can you help me on Hairpinning VPN

  • Bryan

    THANK YOU for this! I had the “same-security-traffic permit” and even the (outside,outside) NAT, and I couldn’t figure out why this wouldn’t work. When I moved NAT exemption to the top of the list, that did it.

  • Steven J. Williams

    I cannot get my client vpn client to L2L traffic flow to work….ideas?