TCP Fast & Curious: A Look At TCP Fast Open

“The TCP is intended to provide a reliable process-to-process communication service in a multinetwork environment. The TCP is intended to be a host-to-host protocol in common use in multiple networks.” – RFC 793

There was a Packet Capture Challenge #5 – TCP Fast Open and Home Routers from CloudShark.  I participated and “won” (they were gracious enough to send me one with my close enough to an answer response) a *P-CAP*.

But more importantly, the challenge got me curious about the TCP standard that was published in 1981, and some of the optimizations past, present and future that it has gone through.

Is It Still Possible To Extend TCP?

Is it still possible to extend TCP?The short answer is that we can still extend TCP, but extensions’ design is very constrained as it needs to take into account prevalent middlebox behaviors. TCP Options are the intended mechanism by which TCP can be extended. Standardized and widely implemented options include Maximum Segment Size (MSS), defined in 1981; Window Scale, defined in 1988; Timestamp, defined in 1992; and Selective Acknowledgment (SACK), defined in 1996. IANA also lists TCP options defined since 1996, but SACK is the most recently defined option in common use, and predates almost all of today’s middleboxes. The question we wish to answer is whether it is still possible to rapidly deploy new TCP functionality using TCP options by upgrades purely at the end systems.”

This post is focused on the IETF Internet draft from Google covering TCP Fast Open.

RFC 793 states that the “three-way handshake” is the procedure for establishing a connection, but it never says it cannot send data until this is complete. This is the normal way for the standard to operate, “Although these examples do not show connection synchronization using data-carrying segments, this is perfectly legitimate, so long as the receiving TCP doesn’t deliver the data to the user until it is clear the data is valid (i.e., the data must be buffered at the receiver until the connection reaches the ESTABLISHED state).”

What Is TCP Fast Open (TFO)?

TCP Fast Open (TFO) is an extension to speed up the opening of successive TCP connections between two endpoints. It works by skipping the three-way handshake, and re-authenticating to the server with a cryptographic “cookie”, stored on the client side.”

TCP Fast Open is described in the IETF Internet draft from Google, and I have quoted pieces of that draft for a brief understanding of the operation as well as added my thoughts and comments.

“TCP Fast Open (TFO) allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, thus providing a saving of up to one full round trip time (RTT) compared to standard TCP requiring a three-way handshake (3WHS) to complete before data can be exchanged.” One important thing to remember about TFO is that the data will need to fit into the SYN packet and will mostly be used for short TCP flows that terminate relatively quickly after handshaking is completed, which is a significant cause of latency for such short flows.

“Network latency is determined by the round-trip time (RTT) and the number of round trips required to transfer application data. RTT consists of transmission delay and propagation delay. Network bandwidth has grown substantially over the past two decades, much reducing the transmission delay, while propagation delay is largely constrained by the speed of light and has remained unchanged. Therefore reducing the number of round trips has become the most effective way to improve the latency of Web applications [CDCM11].” I do all my Internet over Verizon MiFi 3G due to my remote location and satellite just being too slow, also I wonder about the TFO and the overall billing if it would/could same me money by cutting out the 3WHS for sites that I commonly use and refresh on a continuing basis.

Google testing has shown “that TFO improves page load time by 10% to 40%. Only transaction-type applications where RTT constitutes a significant portion of the total end-to-end latency will likely benefit from TFO. Moreover, the client request must fit in the SYN packet. Otherwise there may not be any saving in the total number of round trips required to complete a transaction.”

TFO has similar issues with a previous standard that was proposed and failed due to security issues, T/TCP (RFC 1644). “A past proposal called T/TCP employs a new TCP “TAO” option and connection count to guard against old or duplicate SYNs [RFC1644].” And Google stated these concerns in the draft, “TCP Extensions for Transactions [RFC1644] attempted to bypass the three-way handshake, among other things, hence shared the same goal but also the same set of issues as TFO. It focused most of its effort battling old or duplicate SYNs, but paid no attention to security vulnerabilities it introduced when bypassing 3WHS. Its TAO option and connection count, besides adding complexity, require the server to keep state per remote host, while still leaving it wide open for attacks. It is trivial for an attacker to fake a CC value that will pass the TAO test. Unfortunately, in the end its scheme is still not 100% bullet proof as pointed out by [PHRACK98]. As stated earlier, we take a practical approach to focus TFO on the security aspect, while allowing old, duplicate SYN packets with data after recognizing that 100% TCP semantics is likely infeasible. We believe this approach strikes the right tradeoff, and makes TFO much simpler and more appealing to TCP implementers and users.”

“The key component of TFO is the Fast Open Cookie (cookie), a message authentication code (MAC) tag generated by the server. The client requests a cookie in one regular TCP connection, then uses it for future TCP connections to exchange data during 3WHS.”

TFO Protocol Overview

  Requesting Fast Open Cookie:

  1. The client sends a SYN with a Fast Open Cookie Request option.

  2. The server generates a cookie and sends it through the Fast Open Cookie option of a SYN-ACK packet.

  3. The client caches the cookie for future TCP Fast Open connections (see below).

  Performing TCP Fast Open:

  1. The client sends a SYN with Fast Open Cookie option and data.

  2. The server validates the cookie:

     a. If the cookie is valid, the server sends a SYN-ACK acknowledging both the SYN and the data. The server then delivers the data to the application.

     b. Otherwise, the server drops the data and sends a SYN-ACK acknowledging only the SYN sequence number.

  3. If the server accepts the data in the SYN packet, it may send the response data before the handshake finishes. The max amount is governed by the TCP’s congestion control [RFC5681].

  4. The client sends an ACK acknowledging the SYN and the server data.  If the client’s data is not acknowledged, the client retransmits the data in the ACK packet.

  5. The rest of the connection proceeds like a normal TCP connection.

The client can perform many TFO operations once it acquires a cookie until the cookie is expired by the server. Thus TFO is useful for applications that have temporal locality on client and server connections.

Can We Use TFO Today?

I took a look at TFO on the client side using Linux kernel version 3.6 and Chrome, opening it via the command line with the TFO option, ‘google-chrome –enable-tcp-fastopen’.  Server-side implementation of TFO is not planned until Linux kernel version 3.7.  When opening the Google Chrome browser, I could not get a webpage to load. I could not even see any packets being transmitted when running a Wireshark capture.

Wrapping It Up

TCP and other protocols that we use every day on the Internet can definitely keep you curious when you start digging into how they work. And interoperability issues with new features can and will cause all kinds of issues that will have you digging into RFCs and standards trying to figure out why it is broken, as well as looking for baseline packet captures to compare the working and broken packet flows.

This is one of the most interesting aspects of the challenges that CloudShark provides: a good chance to look at some baseline captures online in a web browser that gets you to digging. My answer to the challenge was that I thought there might be an issue with NAT, even though the draft RFC says it will not break the TCP connection, but just ignore the performance enhancement.

CloudShark provided their answer, which is that when the devices in the path or middleboxes do not understand the TFO option in the TCP SYN packet, the gateway drops the SYN packet and does not allow the connection to complete. I think this is the biggest hurdle to TCP enhancements.

Unknown TCP options are ignored by the receiving host. A TCP extension typically adds a new option to the SYN to request the new behavior. If the SYN/ACK carries the corresponding new option in the response, the new functionality is enabled. Middleboxes have the potential to disrupt this process in many ways, preventing or at least delaying the deployment of new functionality. If a middlebox simply removes an unknown option from the SYN, this should be benign – the new functionality fails to negotiate, but otherwise all is well. However, removing an unknown option from the SYN/ACK may be less benign – the server may think the functionality is negotiated, whereas the client may not. Removing unknown options from data packets, but not removing them from the SYN or SYN/ACK would be extremely problematic: both endpoints would believe the negotiation to use new functionality succeeded, but it would then fail. Finally, any middlebox that crashes, fails to progress the connection, or explicitly resets it would cause significant problems.”

The TFO draft states the issue is about 5% of the devices on the Internet will not forward SYN packets containing unknown options or data. TCP fast open simply will not work in these cases, such as in the CloudShark challenge. The client will have to be aware when this happens and hopefully be able to reset to the ordinary options to continue communications. The “Is it still possible to extend TCP?” paper concludes that approximately 25% of the paths were interfered with in some way beyond basic firewall functions, and that is possible to extend TCP but with some caveats.

All in all, this was a neat challenge and got me to thinking and digging. And it won me a *P-CAP* to boot.


  1. says

    I’m betting the TFO draft underestimates the percentage of devices which won’t forward SYN packets containing unknown options or data. Firewalls are notorious for mauling or dropping packets which don’t conform to expectations. Great article though!

Leave a Reply

Your email address will not be published. Required fields are marked *