What is OTV?
Overlay Transport Virtualization (OTV) is a Cisco-proprietary protocol suite that allows us to extend Layer 2 between datacenters with Layer 3 boundaries in between. It works by encapsulating the L2 packets into L3 multicast packets and sending them out to all other OTV AED’s (Authoritative Edge Devices, used for loop prevention). The legacy OTV implementation uses IP Multicast to accomplish this. Starting in the 3.9S IOS-XE code (released April 2013), we are now able to use an Adjacency Server, and transport OTV traffic across standard IP unicast. (This post covers the legacy Multicast mode)
OTV takes advantage of ISIS as an underlying mechanism to establish ‘router’ adjacencies with everyone else in the Bridge Domain, and uses the ISIS TLV (Type/Length/Value) field to set up a ‘MAC Routing table’ to know what MAC addresses can be found across which AED’s.
OTV is strictly a Layer 2 transport protocol; everything it does is based off the MAC addresses, just like any other switch. Think of an OTV Bridge group as one large virtual Ethernet segment.
OTV currently is only supported by Cisco ASR1000, ASR9000, and Cisco Nexus 7000 series chassis, running IOS-XE or NX-OS. From what I gather, there are no current plans to port OTV code to standard IOS.
How does it work?
First off, a few definitions:
OTV interface – The interface to the L2 domain. This interface is layer 2 only, and is seen by the switch as just another multiport device.
OTV Join Interface – The interface that actually sends out multicast packets, establishes adjacencies, etc.
AED – Authoritative Edge Device, the designated/elected active ‘speaker’ in the L2 domain. This box is the sole speaker/listener at a particular site for OTV. If the primary AED goes offline then you can have another AED-eligible box take over those duties
SSM – Source Specific Multicast, this uses a source and destination pair for multicast routing, not just the destination multicast address
ISIS – Intermediate System to Intermediate System, the underlying routing protocol used in OTV to announce what MAC addresses can be found across what OTV AEDs
OTV Bridge Domain – A grouping of OTV speakers that all agree to share OTV information for a group of VLANs
OTV Site identifier – A grouping of OTV speakers at a particular site. Used to determine if an OTV speaker is talking to the same site, or a remote OTV site. (Mainly used for failure detection)
OTV uses Multicast to find and establish ISIS adjacencies with other OTV speakers. It uses SSM (Source Specific Multicast), IP PIM Sparse/Dense mode, and IP IGMP version 3 to establish a Multicast route table on all boxes at the edges, and on the entire forward/reverse path for the stream. For the current implementation of OTV it is required to have multicast enabled in order to function.
Once we have the multicast route table built, an OTV AED-eligible box sends out an IP IGMP join message stating it wants to listen to anything destined for 220.127.116.11. Then it sends out a SSM packet with its OTV join interface IP as the source, and 18.104.22.168 as the destination. This announces to all OTV boxes that it’s available to establish an ISIS adjacency. When another OTV AED-eligible box sees a hello, he will send a message back to the SSM destination and the two will establish an ISIS adjacency. When all the adjacencies come up, then the next step is electing who the AED at a site is.
An AED, or Authoritative Edge Device, is a loop-prevention mechanism that’s used to ensure a loop-free topology. It does this by sending out a L2 broadcast on the backside to see if any other AED-eligible boxes are on the same L2. This is necessary since OTV does not transport STP packets from one data center to another (this is explicitly denied in the code). If an AED-capable box does not see any other AED-eligible speakers, it assumes the role of AED for the site. If it does detect another AED-eligible box, then he assumes a backup role and learns/listens to ISIS packets, but does not announce anything via ISIS or L2. This is to prevent loops where one box gets a broadcast packet via OTV, broadcasts it out, the L2 broadcast hits the second AED box and goes back into OTV, and creates a loop. In the event an AED loses all ISIS adjacencies outside his OTV site ID, it assumes a network event happened and he removes himself as an AED and allows someone else to take over.
Now that the foundation is laid, we can see the magic of OTV. Let’s say we have the following:
Let’s say host 1 needs to talk to host 2 (assume both boxes have no prior knowledge of each other).
- Host1 sends out an IP ARP broadcast for Host2’s IP
- That broadcast gets sent to all ports (except the one it received it on), and hits the OTV interface on Site1 OTV Box 1.
- Site 1 OTV box knows he’s the AED, so he builds in his local table that Host1’s MAC can be found across his OTV interface. He then announces via ISIS that Host1’s MAC is at Site1.
- Site 2 gets this information through ISIS, and populates his MAC routing table (yeah I know, sounds weird).
- Site 1 sends the broadcast to Site 2 through OTV.
- Site 2 sends the packet out its OTV interface as a standard L2 broadcast, using Host1’s MAC as the source (just like any other switch would do).
- Site 2 Layer 2 populates his CAM tables that host1’s MAC is across the L2 trunk to Site2’s OTV box
- Host2 gets the IP ARP request, sends out an IP ARP reply to Host1’s MAC address.
- Site 2 L2 forwards that reply back to the Site2 OTV box.
- Site 2 OTV box receives the reply, and populates in his local table that the MAC for Host2 is local to him.
- Site 2 then announces via ISIS that Host2’s MAC can be found through him. This lets everyone else in the Bridge domain know this information.
- Site 2 then sends the ARP reply to Site 1 OTV box.
- Site 1 OTV box receives the OTV packet, sends it out as an IP unicast to the Site1 L2 domain.
- Site 1 Layer 2 learns that Host1’s MAC can be found out the port going towards the Site1 OTV box.
- Unicast packet then hits Host1 and Host1 learns of Host2’s MAC.
- Traffic then can flow between the boxes seamlessly.
At this point, Site 1 and Site 2 Layer 2 know the MAC address for the respective box is out their L2 port going towards the OTV, and all the OTV speakers know what actual L3 OTV boxes are responsible for which MAC addresses. From the L2 perspective on each end, they have no knowledge of the fact that OTV even exists. As far as they are concerned, the OTV boxes are just transparent bridges.
That covers the basics of Legacy OTV (Multicast) mode. Next up will be the basic setup, validation, and troubleshooting of OTV in a Multicast setup.