TCP Over IP Bandwidth Overhead

How long will it take to transfer a 100MB file over an IPSec tunnel running across a dedicated 100Mbps Ethernet link? 1 Second? Fail! 8s? You’re getting warmer. It’s almost 8.5s without the IPSec and over 9s with it. What’s the big deal with a 1s difference? Well, extrapolate that increase, let’s say it’s 13%, and a file transfer that should take an hour actually takes around 1hr 8 minutes in ideal conditions, ignoring application overhead. To put it another way, you’ve lost 13% of your bandwidth on a good day, just on network overheads.

You might think that these matters are pointless in our high bandwidth world but lets not forget that for the global majority, internet connectivity still looks like a 9600 modem at best. All these bits and Bytes matter to Facebook and Google too, when you’re dealing with hyper-scale connection and data rates, shaving off even 0.5% becomes a big deal; SPDY is a good example of efforts to help with this.

Also, when I want to understand something, I really want to understand it. When someone technical complains about network performance, I like to have some real figures ready as to why they will never, ever get 100Mb of data down that Fast Ethernet link in 1 second. It fills me with confidence knowing the answers I provide to questions of bandwidth and performance are not based on hypothetical fantasy (albiet still caveated with “plus application overheads”).

By the way, if you’re wondering why you can’t shove that 1GB file over a 1Gb link at a decent speed, that’s a whole other ballgame, very well explained by Brad Hedlund here.

Skip to the end for the summary if you really can’t face the math.

appointment-soonAssumptions

The following assumptions have been made when formulating these calculations;

  • No retries, packet losses or other events occur
  • One way, one to one host communication data and overhead
  • An IP MTU of 1500Bytes
  • A host TCP MSS of 1460Bytes
  • TCP & IP packet headers of 40Bytes (no TCP options)
  • Use of a full duplex communications medium, i.e. the full bandwidth is available both upstream and downstream, at the same time
  • Where division of the data into the maximum packet size results in a fraction, a packet for that remaining data must still be transmitted so we always round up when calculating the number of packets

Units of Measurement

The abbreviations used with the data/traffic values in this article are metric prefixes (aka SI prefixes) indicating decimal multiplication (rather than binary prefixes where 1k = 1024), as follows;

  • 1 Byte = 8 bits
  • 1 kB = 1,000 Bytes (8,000 bits)
  • 1 MB = 1,000,000 Bytes (8,000,000 bits)

A few things to remember;

  • Serial line speeds are typically quotes using metrix prefixes, so a 2Mb E1 is actually 2.048Mb using SI prefixes
  • File sizes are normally quotes in Bytes
  • Link speeds are quoted in bits per second, not Bytes, thus (ignoring all overheads) it’ll take 8s to move 100MB (MegaByte) over a 100Mb (MegaBit) per second Fast Ethernet link
  • Linux commands display file size information in Bytes using SI prefixes. However, using  a –human-readable or -h switch normally results in output using metric prefixes which is rather confusing
  • Windows displays file sizes using metric prefixes.  To make accurate calculations, view the file properties or use a command prompt to discover the file size in bytes and apply the metric prefix
  • Most other storage systems use binary prefixes
  • But hard drive manufacturers typically use SI prefixes which means they appear smaller than specified when capacity is displayed using binary prefixes

Other Overheads

TCP/IP handshake overheads have been ignored as these are negligible.

Ethernet transmission overheads have been ignored too (for now) but I’ll borrow some figures from Wikipedia in the summary to demonstrate how you can easily calculate that affect too.

For the sake of simplicity I’ve ignored Ethernet’s preamble, start frame delimeter and interpacket gap when calculating it’s overhead.

Overhead Calculations

I’ve used five data sizes here to demonstrate that the overhead remains fairly consistent as long as the amount of data exceeds the common IP MSS of 1460 Bytes;

1B of Data

This might seem unlikely but programs such as Telnet and SSH transmit a packet for every character sent or received during a session.

  • 1B can be contained in 1 packet not exceeding 1460Bytes (the default TCP MSS)
  • 1 x 40Bytes of TCP & IP headers equals a 40Byte, 4,100% TCP/IP overhead
  • Thus, 41Bytes of data is actually transmitted over the network

1kB of Data

  • 1kB (1,000Bytes) can be contained in 1 packet not exceeding 1460Bytes (1,000 / 1460 = 0.684.)
  • 1 x 40Bytes of TCP & IP headers equals a 40Byte, 4% TCP/IP overhead
  • Thus, 1,040Bytes of data is actually transmitted over the network

20kB of Data

  • 20kB (20,000Bytes) must be split into 14 packets, each packet not exceeding 1460Bytes (20,000 / 1460 = 13.70.)
  • 14 x 40Bytes of TCP & IP headers equals a 560Byte, 2.8% TCP/IP overhead
  • Thus, 20,560Btyes of data is actually transmitted over the network

480kB of Data

  • 480kB (480,000Bytes) must be split into 329 packets, each packet not exceeding 1460Bytes (480,000 / 1460 = 328.77.)
  • 329 x 40Bytes of TCP & IP headers equals a 13160Byte, 2.74% TCP/IP overhead
  • Thus, 493,160Bytes of data is actually transmitted over the network

1MB of Data

  • 1MB (1,000,000Bytes) must be split into 685 packets, each packet not exceeding 1460Bytes (1,000,000 / 1460 = 684.93.)
  • 685 x 40Bytes of TCP & IP headers equals a 27,400Byte, 2.74% TCP/IP overhead
  • Thus, 1,027,400Bytes of data is actually transmitted over the network

Summary

So, as demonstrated, for data payloads in excess of the common TCP payload maximum segment size (the MSS) of 1460 Bytes, the TCP over IP bandwidth overhead is approximately 2.8%. This equates to an ‘efficiency’ of 97.33% (1460/1500) – in other words, that’s how much bandwidth is left for actual data if you’re putting as much data in each packet as possible.

However, keep in mind that for very small data payloads (common with applications such as Telnet, TN3270 mainframe emulation and SSH) the TCP over IP bandwidth overhead can be higher than 4,000%.

If you add Ethernet (and VLAN tagging) into the mix (see the calculations from Wikipedia here) then the throughput of a 100Mb link is 100 X 0.9733 (TCP/IP efficiency) x 0.9728 (Ethernet (with tagging) efficiency) which equals 94.68Mbps, which I assume means the combined efficiency is 94.68%.

Add in your security protocol (AES would drop the figure to 87.7%), session and application overheads and you start to see where all your precious bandwidth is going, if it is precious. If you’re not filling every packet to the brim you can imagine the loss can get even higher pretty quick.

With HTTP for instance, a large number of headers, large cookie values, multiple cookies and the like can literally consume a further 25% of your bandwidth without you even trying, or realising.

If you are looking for an argument as to why you should use Jumbo frames in a LAN environment, I don’t think there is one; still, like I said, in some circles ever % matters.

Other articles in this series;

The icon Artwork used in this article is by the GNOME Project and licensed under the Creative Commons Attribution-Share Alike 3.0 United States License.

Steven Iveson

Steven Iveson

Steven Iveson, the last of four children of the seventies, was born in London and has never been too far from a shooting, bombing or riot. He's now grateful to live in a small town in East Yorkshire in the north east of England with his wife Sam and their four children. He's worked in the IT industry for over 15 years in a variety of roles, predominantly in data centre environments. Working with switches and routers pretty much from the start he now also has a thirst for application delivery, SDN, virtualisation and related products and technologies. He's published a number of F5 Networks related books and is a regular contributor at DevCentral.
Steven Iveson
Steven Iveson
  • Calvin Christopher

    Excellent article; to be able to explain something to me like I’m five years old is a real art, you’ve accomplished this with a technical topic.
    You have won my “Hero of the Day” award.

    • Steven Iveson

      Thanks Calvin, you’ve made my morning.

    • puzzled

      I can’t tell if you are being sarcastic or not.

      • Steven Iveson

        I wasn’t sure myself but my ego assured me it was positive :)

  • Klovo Alzo

    This is outstanding. Thank you for posting this.

    • Steven Iveson

      Thanks Klovo and you’re welcome, I was rather worried no-one would be interested!

  • What Lies Beneath

    Very true. Thanks for the link, the article is awesome. I’ll add it to the post. Cheers

  • MarvinRhoads

    Thanks for reinforcing to all who read that a bit is not equal to a byte (1 Mbps ≠ 1 MBps).

    I would make it even stronger and reiterate that data storage is measured in bytes and the prefixes based on binary powers of two. Data transmission is measured in bits per second and the prefixes are SI (the latter as you already noted). (8 Mbps ≠ 1 MBps)

    I continue to be amazed at the number of times I need to point this out to my engineering colleagues.

    • Matthias van der Heide

      If only this were true for storage vendors (hard drive manufacturers) as well. They happily continue specifying their products capacities in SI prefixes and bytes, which is confusing and disappointing once you have formatted the 1TB drive and have only 800+ GB of usable space.

      • Steven Iveson

        Indeed. There are inconsistencies all over the place. Again, I’ll expand that part of the article later today. Thanks for commenting.

    • Steven Iveson

      Anytime! I’ll expand on that section later tonight I think. Me too. Cheers

  • jjclee

    Out of interest could you show the maths for the 87.7% IPsec using AES possibly?

    • Steven Iveson

      I can; I’m planning another post on that this week. I’m somewhat less confident of my understanding with that one. Hopefully the public scrutiny will improve that and we’ll all learn something.

  • Andres Velandia

    Thanks a lot for the article. Sometimes I think network administrators should have a solid grasp of concepts like this one; instead, there is a tendency to focus on learning to use a specific tool of one maker on the expense of the always useful foundations. However, I have a question about the remark you made regarding Jumbo frames: Are you implying that jumbo frames present no real benefit in LAN environments?

    • Steven Iveson

      Hey Andres. Thanks for commenting. Well, whilst I make the point that even 1% can matter in terms of performance, in a LAN environment (rather than an internet or WAN link scenario) I don’t see the benefit in saving 1-3% when you’re losing 4-5 on ‘standard’ overheads and way more if security is in play.