When I first looked at the documentation for ERSPAN I could imagine some uses for it. In some cases it could replace RSPAN, but since it’s only available on Cisco Nexus switches, newer Catalyst 6500s, Cisco ASR routers, and other “high end” devices, I determined that it really had limited uses.
But I was wrong. Very wrong. ERSPAN is awesome and in this article, I’ll show you why.
What is ERSPAN?
Here’s a quick overview. ERSPAN is an acronym that stands for encapsulated remote switched port analyzer. ERSPAN mirrors traffic on one or more “source” ports and delivers the mirrored traffic to one or more “destination” ports on another switch. The traffic is encapsulated in generic routing encapsulation (GRE) and is, therefore, routable across a layer 3 network between the “source” switch and the “destination” switch. This works great if you have a dedicated system running a packet sniffer — e.g. Wireshark — connected to an ERSPAN-capable “destination” switch, but what if you don’t?
But There’s an Easier Way . . .
This is where my new favorite trick comes in. When configuring the IP address of the destination, what happens if you simply enter the IP address of your own PC? Yes, all of the encapsulated mirrored traffic is sent to your PC’s IP address. With a simple capture filter setup in Wireshark you can limit your captured packets only to GRE packets. Now you’re only seeing the mirrored traffic. But it gets better. Wireshark is very smart. It realizes that the traffic is encapsulated and automatically displays the “real” source and destination IP addresses of the captured traffic, not the source switch’s IP address and your PC’s (destination) IP address. As a bonus, if you’re sniffing a VLAN trunk, the 802.1Q tags are also captured in the ERSPAN header info.
So How Do I Configure This?
First configure your “source” switch. On a Cisco Nexus 7000 Series switch it looks like this:
monitor session 1 type erspan-source
description ERSPAN direct to Sniffer PC
erspan-id 32 # required, # between 1-1023
vrf default # required
destination ip 10.1.2.3 # IP address of Sniffer PC
source interface port-channel1 both # Port(s) to be sniffed
filter vlan 3900 # limit VLAN(s) (optional)
no shut # enable
monitor erspan origin ip-address 10.1.2.1 global
On your Sniffer PC running Wireshark, you’ll want to configure a Capture Filter that limits the captured traffic to IP Protocol number 47, which is GRE. 47 in HEX is 2F, so the capture filter for this is ip proto 0x2f.
Lastly, start your capture. You should see something like this:
Notice that you can even see the VLAN tag (Vlan: 3900) in the ERSPAN header. This is an advantage over RSPAN, which strips off any 802.1Q tags, because it cannot transport them.
Bonus
This trick works from any ERSPAN-capable switch including all of the Cisco Nexus switches as well as some Catalyst switches and Cisco routers. Here’s the best part: It also works with the Cisco Nexus 1000V.* That’s right. You can capture traffic from a virtual guest OS running in VMWare or HyperV by simply setting the guest OS’s respective Vethernet port as the “source” of your ERSPAN session and your PC’s IP address as the “destination.”
And that’s why ERSPAN is my new favorite packet capturing trick.
Nice write up Gary.
It seems you could easily set up a machine in a management network as the capture device and ERSPAN any device’s traffic under your control back to there.
VMware claims that ERSPAN is also supported in NSX.
Additionally, while Open vSwitch doesn’t support ERSPAN per se, it does support remote packet capture over GRE, which is nearly functionally equivalent. (ERSPAN also has support for remote timestamping – it’s not clear whether this is present in the Open vSwitch implementation.)
Hi Jim,
ERSPAN is actually a function of the Distributed vSwitch thanks to ESX 5.5. Therefore it is supported within NSX due to being baked into the ESX 5.5 version.
Awesome stuff. Great write up.
*Disclaimer – I work for VMware in the Network Security Business Unit (NSX team)
ERSPAN header is not exactly same as L2GRE header. ERSPAN also has span-id in the header. Question on ESX 5.5 – does it do ERSPAN or L2GRE?
ESX 5.5 only uses L2GRE, and does *NOT* support ERSPAN. ESX 6.0 and onwards DOES support ERSPAN which is an extension to GRE and includes Cisco additions such as Vlan tag and Span ID. Which means with 5.5 you cannot mirror packets from VDS to, say, a Cisco router because the Cisco router expects the ERSPAN header. You can however terminate the L2GRE from an ESX 5.5 system on Wireshark, or a Linux box, or certain Cisco IOS “XE”-based products like the ASR 1000 series or the 4500-series.
ERSPAN is amazing but in my labs I couldn’t receive the proper traffic because my swiches uses VPC.
If anyone have a successfully use VPC and RSPAN please share 🙂
In our lab, we have a VPC as well. As a result, we just set up ERSPAN on both members of the VPC so both VPC port channels were covered. Both monitor sessions directed the traffic to the same PC running wireshark. We were able to determine the origin of the traffic based on the source IP ( that is , we knew which monitor session sent the frame to the PC ).
An excellent article, thanks a lot Gary!
If anyone interested in packet capturing by stripping the GRE headers try GULP http://staff.washington.edu/corey/gulp/ (check the documents for 64 bit systems). It needs patching however for VMware, the GRE_HDRLEN definition should be 42 instead of Cisco’s 50 at the top of gulp.c file.
This way you can set up your tcpdump/wireshark capture filters for the real insteresting inner contents.
I have made a patch, which adds a new command line option to set custom header length, so you could use the same binary for Cisco and VMware as well, if anyone is interested I can paste it here.
please paste the patch,thank you.
I’ve searched it for long time.
Here is the patch I made: http://goo.gl/0SkgKA
Besides capturing with Wireshark or stripping the ERSPAN headers with GULP, there’s another tool to (re)process ERSPAN: “rcdcap” (http://sourceforge.net/projects/rcdcap/). RCDCAP can write capture files but it can also strip the ERSPAN header, reconstruct or strip the VLAN header and then copy the traffic to another interface: regular Ethernet, veth pair, etcetera. It also supports HP ProCurve’s proprietary remote span feature based on UDP encapsulation and was designed for extensibility. Other remote capture protocols may be added later.
Just plugging it here, I’ve used it many times after my employer (a Dutch uni) had it written.
my team have a problem of installing this RCDcap on a Linux system, can you give me some tips on installation?
Is there a way to ENCAPSULATE traffic into an ERSPAN tunnel without using a cisco device? What if I wanted to source ERSPAN traffic from a Linux host instead of a switch?
Hi Guys,
for the linux noobs using windows……thats me, i created a short video on how to strip the ERSPAN headers off the packet capture using wireshark natively;
https://www.youtube.com/watch?v=JuZ8bv0qc6E
NICE video!
But how to determine the offset value. In my example “38” seems wrong …
Unfortunately it looks like this works ONLY on the default VDC 🙁
I tried it here but since we are in another VDC this is what i get at the end (very disappointing!):
HOSTNAME(config)# monitor erspan origin ip-address 10.29.92.253
ERROR: Per VDC origin IP not supported. Please use global mode
HOSTNAME(config)# monitor erspan origin ip-address 10.29.92.253 global
ERROR: Global origin IP config possible only in default VDC
Nice chicken and egg situation? Any ideas?
Thanks!
An example Wireshark capture Filter for filtering IP host addresses within an ERSPAN Session from Cisco ACI:
ip proto 0x2f and ((ip[54:4]==0x0A7B7B7B) or (ip[58:4]==0x0A7B7B7B))
0x0A7B7B7B represents an IP address in HEX format. In this case 10.123.123.123
Important: The offset (54 / 58 in my example) can change. E.g ACI is sending a VLAN Tag within the ERSPAN Session. If you mirror traffic without a VLAN tag, you have to lower the numbers by 4. It also changes if there are IP options within the outer IP header. Offset seems to be stable as long as you don’t change the port mirror configuration on your switch. -> Always check the offset before start capturing your data.
Thanks Gary and Marco,
this worked for me with a twist, I used the ether[offset:lenght] filter instead of ip.
Hello
How can i filter the capture with a specific MAC. i want to capture only LACP a L2 protocol
Thanks