How long will it take to transfer a 100MB file over an IPSec tunnel running across a dedicated 100Mbps Ethernet link? 1 Second? Fail! 8s? You’re getting warmer. It’s almost 8.5s without the IPSec and over 9s with it. What’s the big deal with a 1s difference? Well, extrapolate that increase, let’s say it’s 13%, and a file transfer that should take an hour actually takes around 1hr 8 minutes in ideal conditions, ignoring application overhead. To put it another way, you’ve lost 13% of your bandwidth on a good day, just on network overheads.
You might think that these matters are pointless in our high bandwidth world but lets not forget that for the global majority, internet connectivity still looks like a 9600 modem at best. All these bits and Bytes matter to Facebook and Google too, when you’re dealing with hyper-scale connection and data rates, shaving off even 0.5% becomes a big deal; SPDY is a good example of efforts to help with this.
Also, when I want to understand something, I really want to understand it. When someone technical complains about network performance, I like to have some real figures ready as to why they will never, ever get 100Mb of data down that Fast Ethernet link in 1 second. It fills me with confidence knowing the answers I provide to questions of bandwidth and performance are not based on hypothetical fantasy (albiet still caveated with “plus application overheads”).
By the way, if you’re wondering why you can’t shove that 1GB file over a 1Gb link at a decent speed, that’s a whole other ballgame, very well explained by Brad Hedlund here.
Skip to the end for the summary if you really can’t face the math.
The following assumptions have been made when formulating these calculations;
- No retries, packet losses or other events occur
- One way, one to one host communication data and overhead
- An IP MTU of 1500Bytes
- A host TCP MSS of 1460Bytes
- TCP & IP packet headers of 40Bytes (no TCP options)
- Use of a full duplex communications medium, i.e. the full bandwidth is available both upstream and downstream, at the same time
- Where division of the data into the maximum packet size results in a fraction, a packet for that remaining data must still be transmitted so we always round up when calculating the number of packets
Units of Measurement
The abbreviations used with the data/traffic values in this article are metric prefixes (aka SI prefixes) indicating decimal multiplication (rather than binary prefixes (based on powers of two) where 1kbit = 1024bits), as follows;
- 1 kB = 1,000 Bytes (8,000 bits)
- 1 MB = 1,000,000 Bytes (8,000,000 bits)
A few things to remember;
- Serial line speeds are typically quoted using binary prefixes, so a 2Mb E1 is actually 2.048Mb using metric prefixes
- Ethernet speeds are typically quoted using metric prefixes, so a 100Mb Ethernet link is exactly that (100,000,000bit)
- Linux commands display file size information using metric prefixes. However, using a –human-readable or -h switch normally results in output using binary prefixes which is rather confusing
- Windows displays file sizes using binary prefixes. To make accurate calculations, view the file properties or use a command prompt to discover the file size in Bytes and apply the metric prefix
- Most other storage systems use binary prefixes
- Hard drive manufacturers typically use metric prefixes which means they appear smaller than specified when capacity is displayed using binary prefixes
Bits & Bytes
- 1 Byte = 8 bits
- File sizes are normally quoted in Bytes
- Linux commands display file size information in Bytes
- Link speeds are quoted in Mb (Megabits) per second, not MegaBytes, thus (ignoring all overheads) it’ll take 8s to move 100MB (MegaByte) over a 100Mb (Megabit) per second Fast Ethernet link
TCP/IP handshake overheads have been ignored as these are negligible.
Ethernet transmission overheads have been ignored too (for now) but I’ll borrow some figures from Wikipedia in the summary to demonstrate how you can easily calculate that affect too.
For the sake of simplicity I’ve ignored Ethernet’s preamble, start frame delimeter and interpacket gap when calculating it’s overhead.
I’ve used five data sizes here to demonstrate that the overhead remains fairly consistent as long as the amount of data exceeds the common IP MSS of 1460 Bytes;
1B of Data
This might seem unlikely but programs such as Telnet and SSH transmit a packet for every character sent or received during a session.
- 1B can be contained in 1 packet not exceeding 1460Bytes (the default TCP MSS)
- 1 x 40Bytes of TCP & IP headers equals a 40Byte, 4,100% TCP/IP overhead
- Thus, 41Bytes of data is actually transmitted over the network
1kB of Data
- 1kB (1,000Bytes) can be contained in 1 packet not exceeding 1460Bytes (1,000 / 1460 = 0.684.)
- 1 x 40Bytes of TCP & IP headers equals a 40Byte, 4% TCP/IP overhead
- Thus, 1,040Bytes of data is actually transmitted over the network
20kB of Data
- 20kB (20,000Bytes) must be split into 14 packets, each packet not exceeding 1460Bytes (20,000 / 1460 = 13.70.)
- 14 x 40Bytes of TCP & IP headers equals a 560Byte, 2.8% TCP/IP overhead
- Thus, 20,560Btyes of data is actually transmitted over the network
480kB of Data
- 480kB (480,000Bytes) must be split into 329 packets, each packet not exceeding 1460Bytes (480,000 / 1460 = 328.77.)
- 329 x 40Bytes of TCP & IP headers equals a 13160Byte, 2.74% TCP/IP overhead
- Thus, 493,160Bytes of data is actually transmitted over the network
1MB of Data
- 1MB (1,000,000Bytes) must be split into 685 packets, each packet not exceeding 1460Bytes (1,000,000 / 1460 = 684.93.)
- 685 x 40Bytes of TCP & IP headers equals a 27,400Byte, 2.74% TCP/IP overhead
- Thus, 1,027,400Bytes of data is actually transmitted over the network
So, as demonstrated, for data payloads in excess of the common TCP payload maximum segment size (the MSS) of 1460 Bytes, the TCP over IP bandwidth overhead is approximately 2.8%. This equates to an ‘efficiency’ of 97.33% (1460/1500) – in other words, that’s how much bandwidth is left for actual data if you’re putting as much data in each packet as possible.
However, keep in mind that for very small data payloads (common with applications such as Telnet, TN3270 mainframe emulation and SSH) the TCP over IP bandwidth overhead can be higher than 4,000%.
If you add Ethernet (and VLAN tagging) into the mix (see the calculations from Wikipedia here) then the throughput of a 100Mb link is 100 X 0.9733 (TCP/IP efficiency) x 0.9728 (Ethernet (with tagging) efficiency) which equals 94.68Mbps, which I assume means the combined efficiency is 94.68%.
Add in your security protocol (AES would drop the figure to 87.7%), session and application overheads and you start to see where all your precious bandwidth is going, if it is precious. If you’re not filling every packet to the brim you can imagine the loss can get even higher pretty quick.
With HTTP for instance, a large number of headers, large cookie values, multiple cookies and the like can literally consume a further 25% of your bandwidth without you even trying, or realising.
If you are looking for an argument as to why you should use Jumbo frames in a LAN environment, I don’t think there is one; still, like I said, in some circles ever % matters.
Other articles in this series;