The number of overlay technologies available today for the datacenter are numerous and highly functional. The flexibility they provide enables security zone enforcement and physical portability of hosts more seamlessly (among other benefits). However, a few risks in deploying popular layer 2 overlay technologies are vendor-lockdown, scalability, specialized hardware required to mitigate bottleneck points, and predictability of traffic load. These were enough concerns to drive me to come up with the solution I present here in my first Packet Pushers blog.
In addition to the usual goals of availability and scalability, the specific aims of this design are: 1. Enforcing firewall security zones in a layer 3 environment, and 2. Isolating each layer 2 environment to one or two switches at most.
Very often, once a firewall is placed in the datacenter network, each firewall interface/zone is associated with one VLAN, and the hosts sit in that VLAN. The firewall serves as the default gateway. The diagram below illustrates a sample of this typical “layer 2” network (redundancy of firewalls, core switches are omitted for simplicity).
The flexibility of this design can be seen by the unstructured placement of servers. Physical link speed requirements notwithstanding, a server can be placed in any rack and will have connectivity to their VLAN. This is especially important in a dynamic VM environment. However, as the environment grows, size of the switched network (think: spanning-tree) can become a risk. Mac address table size and arp table size on the firewall should also prompt concern if 1000+ interfaces are possible in your network.
These classic problems are what drive a Layer 3 discussion. But too often a Layer 3 design restricts placement of servers to be connected to a purpose-reserved “core.” Other times it’s too difficult to ensure all traffic goes through the firewall, and so you wind up with multiple firewalls (ugh), access-lists on routers (sigh), or omission of security policy enforcement on the network (here comes my CSO). Using VRF’s, however, on a capable router or layer 3 switch attached to your firewall, we can overcome these issues.
Starting with our Layer 2 network design pictured above, we replace the “Big Core Switch” with a router (practically speaking, if you already have a Big Core Switch that can handle VRF’s, routing protocols, and a lot of ARP, you don’t need to change hardware). Then we setup sub-interfaces on the link between the firewall and router – one sub-interface per security zone. On the router side, we assign each sub-interface to a VRF.
At this point, the firewall and router share one physical link, but logically, the firewall has three connected subnets, each shared with a different routing-instance, or virtual router.
Next, we extend this sub-interface scheme further downstream – between the router and access switches:
Notice again, there’s a single physical point-to-point between the router and neighboring device (in this case, Access Switch), but we create sub-interfaces on the router to correspond with different VLANs on the downstream switches. I’ve included a suggested subnet scheme. With this layout, the zones, subnets and VLANs all conveniently agree on numbers.
10.10.0.0/16 – dmz: p2p vlan 10, server vlans 100
10.20.0.0/16 – apps: p2p vlan 20, server vlans 200
10.30.0.0/16 – mgmt: p2p vlan 30, server vlans 300
Notes about the example:
- I’m using interfaces and configurations from Juniper platforms in this article, but the concept can be applied to Cisco platforms with the same supported features.
- For simplicity in explaining the technique, I have not included redundant components of this design, however, each area can be made redundant. At the router level, point-to-point connectivity between routers requires a sub-interface per VRF, and a routing protocol is advised.
Another way to look at the resulting network is in this Layer 3 “logical” view:
And because we’ve aggregated subnets by security zone AND routing domain, our route table on the firewall for all the downstream subnets is incredibly simple (albeit idealistic):
[ edit routing-options static ]
route 10.10.0.0/16 next-hop 10.10.0.5
route 10.20.0.0/16 next-hop 10.20.0.5
route 10.30.0.0/16 next-hop 10.30.0.5
Of course a routing protocol could be used, and I’d encourage you to do so. The potential for tidy area-range configurations should be obvious.
So what have we gained by doing this? The benefit of these combined features:
- Minimally-sized layer 2 domains, isolating potential bridging problems to a single switch
- Flexibility to connect a host to any switch (in other words, placed in any cabinet)
- Packet flow / Security zone enforcement
- Single routing-table on the firewall / Simplified config of the firewall
- Leveraging a high-throughput firewall without concern for its ARP learning rate or capacity
Adopting this kind of design for your datacenter network might seem too complicated for a network of say, fewer than a hundred hosts. But given the near inevitability of a network’s growth (or sprawl), here’s a ready-to-go, no gimmicks option to ensure that growth aligns with the original master plan.