Most providers — transit, edge, and content — are pretty obvious to the various users of the Internet. Users interact with edge and content providers every day, and transit providers have such large names in the industry that they’re often the subject of news and other articles. But who actually connects all these different providers together?
Private peering is the most common solution — two providers decide to connect at one of the provider’s facilities, so a set of links are run and configured, and BGP peering is configured to exchange the right routes based on the type of peering arrangement. Private peering can, however, have some downsides. Running the links and facilities can be expensive. Once the peering infrastructure is in place, it’s difficult, time consuming, and expensive to switch upstreams (if you’re a customer looking for transit) or to modify other peering arrangements — the cost of changing is similar to the cost of moving in trying to make money in real estate. The sunk costs can often make it difficult to justify changing providers based on cost savings or service improvement.
In some areas, the available upstream providers might only peer in a distant location — for instance, much of the traffic traveling between any two cities in Latin America actually passes through Miami simply because that’s where all the larger ISPs offering transit service locate their peering points. National borders and geographic conditions can dictate peering points that are outside the optimal in terms of optimizing traffic flow.
Locally situated Internet Exchange Points (or Providers — in either case, IXPs) can provide a “short cut” to resolve many of these problems. An IXP is essentially a data center for peering routers. There is physical space to install peering routers, power, cooling, a fabric (either Ethernet or IP based) to which all the peering routers connect, and a (normally) a route server that provides a mechanism for all the routers located in the IXP to share BGP routing tables. Most IXPs operate as a sort-of local cooperative; a lot of providers get together and build an IXP rather than building individual peering points between each pair of providers. This saves money by combining the infrastructure of what would be a larger set of physical interconnection points, and allows each provider to build relationships on an “on demand” basis within the IXP.
IXPs sell their services based on a port — the larger the port onto the fabric you purchase, the more you pay the IXP to collocate your equipment there. Some IXPs charge a “base rate,” for collocation and some smaller sized port, and then charge additional fees for larger ports, but it’s the same basic concept.
Connecting to the IXP fabric doesn’t allow you to send (or receive) traffic from anyone else connected to the fabric. Instead, once connected you still need to negotiate the type and level of peering you want to receive from the other networks connected to the IXP. Connecting to the IXP allows you to peer with a greater number of other providers at a lower cost because you’re only paying for ports in the IXP, and it allows you to modify your peering arrangements more quickly. Changing from one upstream to another is a matter of adjusting policies, rather than building out a new circuit to the new provider, and abandoning the infrastructure put in place for the previous connection. This allows customers to choose between multiple upstreams, and each upstream to sell their services to a broader array of customers in a “public marketplace.” In this sense, an IXP can be considered a sort of “bandwidth bazaar.”
As much as the physical collocation of a lot of networks in a single facility helps to save on infrastructure costs, the real magic is in the route server. Connecting to an IXP with a thousand other providers or networks doesn’t actually mean you need to build a BGP session with each one you’d like peer with. Instead, your BGP speaker can peer with just the route server (or two route servers, in some cases). The route server uses a set of fancy filters and mechanisms to enforce the sending of routes only between networks that have a peering relationship. The complexity of building the peering relationships, filters, and policies are pushed into the IXP fabric, rather than being managed by each connecting provider. Call it “peering as a service,” interconnection as a cloud based service.
IXPs, then, can provider cheaper peering costs, a marketplace of peering, simpler configuration and management for the providers, lower latency, better bandwidth, and a number of other advantages to providers and customers. The main drawback is the IXP does represent a middleman in the relationship — whether or not it’s worth paying this middle man is often determined by specific, local, conditions.
A few good IXPs to look at to get a better feel for fees and services are Midwest-IX, LINX, and Equinix. This recent thread on NANOG’s mailing list is a useful place to start to understand pricing trends and thoughts on peering directly versus through an IXP. draft-ietf-grow-ix-bgp-route-server-operations provides a good overview and best practices for BGP route server operation.