Understanding Brocade’s ISLs and ECMP Just a Wee Bit More

11 May 2012 by 6 Comments

In the Brocade Virtual Symposium hosted by Packet Pushers and Tech Field Day, Part 3: Multi-Path vs. Multi-Chassis shows us an interesting case of using ECMP and VCS with different size ISL links. Ivan Pepelnjak asks a question. If Brocade is using ECMP as dictated by TRILL, will there be underutilised bandwidth as each of the links have the same cost? I feel the answer was not made clear, so I want to explain it more fully.

Let’s take the simplest example of this – just two switches in a VCS fabric, with two ports groups per switches in use for ISLs.

We can see that we have two ISLs, and ECMP defined cost is based on speed and not bandwidth. Both ISL’s are a cost of 500. At this point as far as TRILL/IS-IS standards are concerned, we are done. We have two paths, and we can do flow-based load balancing via hashes across these two paths. What Brocade has done beyond this is to add frame-based load balancing across ISL Link 1 – not flow based. This was covered well in the Brocade Virtual Symposium.

Now the point I want to make (and to answer Ivan’s question – more fully I hope), really is this: since Brocade is using FSPF for both FCoE and TRILL, that ALSO takes into account the bandwidth of the ISLs. This means in the above example that on a flow-based load balancing, Brocade will take the total bandwidth of both ISL’s – 40G in this case – and balance the flows in a 3:1 ratio. Say 8 flows come in. ISL 1 will handle (and then frame-based load balance) 6 flows, while ISL 2 will take the remaining 2 flows.

I would also like to say that these features have been in VDX products since they were launched. So, this is not new anymore and came from the FC side a long time before that.

What happens when vendor interoperability of TRILL arrives? My guess is that Brocade will continue to do load balancing their own way for their own equipment (as this offers a much better use of the links), but will offer a standards-based TRILL port running IS-IS to create non Brocade ISLs.

Thanks

Michael.

Filed Under: Blogs, Switching Tagged With: , , , , , ,
About Michael Schipp

Michael Schipp - @maschipp on Twitter. An IP professional with 10+ years of enterprise networking experience. Working for A10 Networks in Australia. *My opinions only*

  • http://twitter.com/IPv6Freely Chris Jones

    I’m doing a very large VDX deployment right now (I’m being told it’s the largest in North America, but I have a hard time believing that). 

    From what I’ve been told, each TRUNK (one or more links within the same port group between two switches) load balances per packet on each ISL within that trunk. BETWEEN trunks, the load balancing is per-flow.

    I’ve also been told that oddly enough, when we’re talking alternate paths (as in, not between the same two switches), effective hop count actually takes precedence over accumulative cost. I’m not sure how correct this is, and if anyone knows for sure, please reply!

    • http://twitter.com/maschipp Michael Schipp

      Hi Chris,
         Thanks for the reply. I have seen the VDX’s setup this way for myself and working as I explained. Also this is covered in the BCEFE course that I did over a year ago.

      Not sure I understand by – “From what I’ve been told, each TRUNK (one or more links within the same port group between two switches) load balances per packet on each ISL within that trunk. BETWEEN trunks, the load balancing is per-flow.”

      As an ISL is a link between two VDX switches in the same port group of up to 8 individual links. So from the above example we have two ISL’s and flows are split in 3:1 – 3 out of 4 flows are directed to ISL 1 and each of the 3 flows will be frame  load balanced.

      I wish you the best of for your deployment – can you say how many VDX’s you have?

      Thanks
      Michael.

    • http://twitter.com/maschipp Michael Schipp

      Hi Chris,
        I have just reread the training doco for hop count. 

      The accumulative cost is the metric used to select the path.

      The hop count is used to prevent loops – e.g. when a rbridge receives a frame with a HC of zero it will drop the frame.  

      The ingress RBridhe sets the HC to be in excess of the maximum hops it expects to get to the egress RBridge to allow for alt paths.

      Thanks
      Michael.

  • http://twitter.com/tbourke tbourke

    My understanding of how Brocade combines links is they don’t do per-flow load balancing, but actually combine them (and still provide in-order delivery through some sort of mechanism I’m not familiar with) so it is actually a 30 Gbit link, so the cost wouldn’t be the same as the 10 Gbit link.

    • http://twitter.com/maschipp Michael Schipp

      Hi Tony,
          For the purpose of ECMP the link cost is based on speed of the link and not the bandwidth. However Brocade do take into account the bandwidth to allocate a higher weight to an ISL that has more bandwidth.  This is why we see the flows in the example above split into a 3:1.  

      Thanks
      Michael.

  • http://twitter.com/jtarrioBRCD Juan Tarrio

    Whoops, I thought it was clear by my Twitter handle, I’ll make sure to disclose it more visibly next time ;) . Thanks.