Iljitsch van Beijnum, Network Engineer and author, is our guest blogger today with a post about peering sponsored by Noction Intelligent Routing Platform.
Most farmers most of the time sell their produce to super market chains. Most consumers most of the time buy their produce from super market chains. Alternatively, some farmers bring their produce to a farmer’s market where they sell it directly to consumers, cutting out the middleman. The internet business works largely the same way: traditionally, content providers and “eyeball” networks (ISPs serving consumers) each buy transit service from larger networks. But why pay another network when you can exchange the traffic directly?
That’s what we call “peering” in our business. Unlike in the telephony world, where nearly all interconnection involves some form of payment, in the internet business peering is usually settlement-free, also known as sender keeps all (the money paid by their customer). This model wouldn’t work in the telephony world because then a telco with customers that only receive incoming calls wouldn’t make any money. But ISPs charge their customers for both incoming and outgoing traffic, so the business case still works without interconnection fees.
So how does peering work in practice?
There are two flavors: direct or private peering, or peering over a (public) internet exchange (IX). In theory, private peering can happen over a long distance circuit, but most of the time, it happens within the same building, or at least the same city. Private peering really only makes sense when there’s a lot of traffic involved, because each peer requires a separate router or switch port and a circuit. Even within the same building a cable to a peering partner can be costly, as carrier facility operators have discovered that charging for cross connects can be lucrative business.
Peering over an internet exchange requires a connection to the IX, which is basically a large Ethernet switch or set of Ethernet switches. Every IX member/customer connects a router to the IX peering LAN. All these routers get an IP address in the same subnet, so they can talk directly to each other. So the only thing required to let peering traffic flow between two IX members is a BGP session between their respective routers.
However, no matter how polite your peering requests are, some people will say “no”. For instance, ISPs that operate a global network typically aren’t willing to peer with small regional networks: the global ISP would end up transporting the traffic most of the way without getting paid for it. So before connecting to an IX or setting up private peering, it’s essential to get a good estimate of how much traffic you’ll be able to handle over a public or private peering connection, so you can calculate whether the money you save on transit costs will pay for the peering costs. For private peering within the same building, this can be as little as an unused router or switch port and the costs of a cross connect, but peering can get rather expensive if you need to lease a circuit to a remote city and pay a possibly steep IX connection fee.
Also don’t underestimate the work required: connecting to a transit ISP or an IX takes roughly the same amount of work. However, once a transit connection is up, it requires very little attention. With an IX connection, the real work is setting up peering with other networks connected to the IX. It’s hard to quantify the amount of work involved, but a number like 20 minutes to set up a typical peering session and then one minute per month for maintenance is a reasonable estimate. Multiply this by 100 peers and that’s four days full time to set up the peerings and then two hours a month for maintenance.
So in order to determine whether peering makes sense compared to buying transit, we first need to figure out the total cost: hardware, circuits, port and member fees and labor. In the US, the largest exchanges are the Equinix and Any2 IXes in places like New York, Ashburn, Los Angeles, the Bay Area and Miami, and the NYIIX in New York. The three biggest IXes in Europe (and the world) are AMS-IX Amsterdam, DE-CIX Frankfurt and LINX London. Their average price is $650 per month for a 1 Gbps connection and $1900 for a 10 Gbps connection. The bad news is that you pay this amount even if you don’t send a single packet over your connection, but the good news is that you don’t pay any extra even if you fully utilize the connection. For simplicity, let’s assume that the total cost of connecting on an IX are $1000 at 1 Gbps and $2500 at 10 Gbps.
Whether it makes sense financially to get a connection to an IX and peer depends on the amount of money you can save on transit costs. Transit prices vary widely by region, on- or off-net location and volume commitment. If the marginal cost (just the variable part of the total costs) of a megabit of transit is $1, then it doesn’t make sense to get a 1 Gbps IX connection for $1000, as the best you could hope to do is reduce transit costs by the same $1000, and in practice you can’t use a connection at 100%. You’d have to use a 10 Gbps IX connection at 2500 Mbps to break even. However, if your transit costs are $10 per megabit, a 1 Gbps IX connection would start making sense starting at 100 Mbps.
It’s hard to determine in advance how much traffic you can exchange through peering. Many internet exchanges have a route server. By peering with the route server, you automatically peer with everyone else who peers with the route server. You can ask a network similar to you how much traffic (as a percentage of their total traffic) they exchange with route server peers to get an idea of how much you can expect to exchange if you join.
Then, look at the published peering policies of prospective peers, for instance, in PeeringDB. You can expect to peer with most of the networks that say they have an open peering policy, but sometimes this information is out of date, or peering requests go unanswered. You’ll probably want to ask the largest prospective peers if they want to peer with you before committing financially. Don’t be afraid to ask, even if their peering policy is “selective” or “restrictive”, unless they have a published peering policy with requirements that you clearly don’t meet.
Obviously, networks that mainly host content and thus mostly have outbound traffic primarily benefit from peering with networks that mostly connect consumers and thus mostly have incoming traffic, and the other way around. If traffic is mostly in one direction, there’s little benefit in peering with a network with similar traffic patters. However, some larger networks require that incoming and outgoing traffic is roughly balanced in order to set up peering.
Once you have a list of networks that you expect to peer with, you need to figure out how much traffic you’ll be sending to and receiving from those networks. There’s really only one way to do this: with Netflow. Netflow is a system that lets routers sample their traffic. The samples can then be analyzed in various ways, including determining the amount of traffic to/from a given autonomous system.
The calculations for private peering are much the same as for peering over an IX, but with the advantage that you know exactly who you’ll be peering with in advance, so there’s less uncertainty. Despite the fact that there are no IX port costs involved, private peering usually only makes sense between relatively large networks; with smaller networks, there’s just not enough traffic between the two networks to bother.
If you’re not quite ready to start peering yourself, but you do have a significant amount of traffic that could be handled regionally through peering, there may be opportunities to negotiate a discount transit rate for that regional traffic. In such a case, the ISP doesn’t provide transit to all the 500,000 (and counting) IPv4 prefixes, but only to those that they have access to through their own peering. Because they don’t have to pay a bigger ISP for transit or deliver the traffic over long distances, such partial transit can be much cheaper than regular transit service.
Whatever your final peering blend consists of, you must make sure that the proper policies are in place to efficiently route the traffic across your providers and peers. Running BGP buys a lot of peace of mind. However, BGP routing as well as the internal (OSPF) routing and Ethernet spanning tree (and the interactions between them!) add a lot of complexity. So sometimes, these protocols don’t repair connectivity issues the way they should. Or worse, they create outages of their own that wouldn’t have occurred in a simpler network.
Most of the time, BGP will route around outages. However, sometimes packets disappear into a black hole: for some reason, the packets are lost, but the routing protocols don’t notice that there’s a problem, so traffic continues to flow towards the black hole rather than be rerouted.
One way to detect this problem is to monitor reachability of key remote services. Another is looking at total traffic, which will be much lower than usual in the presence of a black hole. Once detected, recovering from routing black hole affecting one ISP/peer is very simple: shut down the BGP session towards that provider until they’ve fixed the problem. Alternatively you can automate the process of selecting the best performing transit provider or peer by deploying a route optimizer (Noction Intelligent Routing Platform) which evaluates all your ISPs, IXes, and partial peers in terms of packet loss and latency and automatically reroutes traffic through the most reliable path.