I ran into an issue over the weekend where a VPN client was unable to access a remote office connected via an L2L tunnel terminated on the same firewall. The symptoms were straightforward enough. The client was unable to either ping or open a URL at a specific server at the remote office, although this connectivity used to work. In this example, VPN client 192.168.100.100 was not able to access server 10.11.12.1, although access to resources in the 10.10.0.0/16 network was fine.
I confirmed the remote office firewall was unlikely to be the issue; the remote firewall had seen no changes. As I knew the headquarters Cisco ASA firewall HAD seen a few changes, that’s where I focused my attention. After reviewing the headquarters firewall rulebase, I knew that the VPN client IP pool had permission to access resources in the remote office.
Monitoring the firewall logs, I spotted several “110003: Routing failed to locate next-hop for protocol from src interface:src IP/src port to dest interface:dest IP/dest port” messages tied directly to the VPN client trying to open a socket to 10.11.12.1. So, I reviewed the firewall routing table with “show route” and “show asp table routing” and found no issues…not that I expected to. If the routing table was having a problem, connectivity issues would have been more widespread.
Of course, NAT sprung to mind as a potential issue, but I couldn’t see an obvious problem. There was a NAT that exempted the entire VPN client pool from being translated to any RFC1918 destinations. As this clearly covered the remote office IP range, I was a little stumped. This confusion was compounded by the fact that the connectivity used to work. A perplexing issue.
After a little googling around, it seemed dead certain that the issue was NAT, although there was no one single obvious solution anywhere that I ran across. XKCD knows where I was.
I felt a little chastened, as I tend to go for the documentation first anyway, although in fairness I did search cisco.com – I just hadn’t found any answers. But with Google being less than helpful, I went back to the Cisco docs again, where with great joy, I found The Answer. Answers are good. Answers make me happy.
Take a read here. A couple of highlights caught my eye:
- The “same-security-traffic permit intra-interface” is required. Fair enough, easy to implement, makes sense, and I’d already done that. No problem.
- Now the documentation got confusing because of two conflicting statements:
- “When the ASA sends encrypted VPN traffic back out this same interface NAT is optional. The VPN-to-VPN hairpinning works with or without NAT.” Okay – so I don’t need to write a NAT statement for the hairpinned traffic. NAT is optional, right? But then you next read…
- “To exempt the VPN-to-VPN traffic from NAT, add commands that implement NAT exemption for VPN-to-VPN traffic.” Uh, hang on. So I *do* need a NAT statement?
From my experience, I believe that, yes, you need a NAT exemption statement. I think all Cisco is trying to say is that you don’t have to actually translate the source or destination address into something else to be able to get through the hairpin.
Writing a NAT exemption statement is not an unusual thing to have to do in an ASA, but the magic in the context of hairpinning is in defining the ingress and egress interfaces. In a hairpin path, the traffic flows in and out the same interface. While I did have a NAT statement that matched source and destination addresses in question, the interfaces were only suitable for handling source VPN client to destination headquarters network traffic…not traffic headed from VPN client to the remote office network. Therefore, I needed a NAT statement like this: “nat (outside,outside) source static client_vpn_pool client_vpn_pool destination static remote_office_net remote_office_net“.
The order of the NAT statement also mattered, as NAT statements are processed in order. Once I moved my new NAT statement to the top of the list, the issue was resolved.
Now…this all left me with some questions which I’m planning to test in a lab to see verify ASA behavior under 8.3 and 8.4. More posts to come on ASA NAT, and your comments welcome if you have experiences of your own to share on ASA NAT under the new paradigm.
