Using SSL Intercept With ADCs + Firewalls to Inspect & Clean Encrypted Traffic

26 January 2013 by 12 Comments

The Problem

Let’s take a typical enterprise. We have our internet connection going to our router (or ISP’s router), then handed off to our firewalls. We then break out to our internal and DMZ networks. Pretty standard stuff. We have our perimeter security controls sitting on the firewalls with a nice rule of “allow TCP port 443 (HTTPS) any to any.”

Now, we all know HTTPS is a secure protocol using SSL/TLS. Well, not quite secure. HTTPS is encrypted, sure, but what has been encrypted? Do our firewalls check the content of the encrypted traffic? Not by default, and in most cases, this is not even possible on firewalls. Our firewalls can do a great job of checking HTTP and other non encrypted protocols, but most will not or cannot inspect encrypted traffic (not without a reduction in performance, if they can do it at all). Stop and have a think about what could be hiding inside of HTTPS, running over your networks and on your PCs.

Is malware really an issue over HTTPS? Have a look at this blog post for some information that indicates this was a problem all the way back in 2008. This threat area has only been growing since then.

The Solution

What can be done to look inside of encrypted packets so that our firewalls can do their job? The answer is SSL intercept. SSL Intercept (or SSL forward proxy) provides a way to inspect encrypted traffic. There are a few vendors that can do this. Your current firewall might be able to do this; Palo Alto Networks and Watchguard are two I know of that can. Some Application Delivery Controllers (ADCs) can also provide this function; A10 Networks and F5 are two that I know of. Be aware that when enabling SSL Intercept on firewalls, you will experience a big drop in performance on platforms that do not have dedicated hardware for encryption and decryption. This is where using ADC’s can offer better performance, as they are designed with SSL offload as a standard feature.

The Implementation

How can we implement SSL Intercept using ADCs and firewalls? The idea is to place SSL intercept devices around your firewalls if…

  • they do not support SSL Intercept or
  • if you want to offload the task of decryption and re-encryption from the firewalls.

In this way, we allow firewalls that can do perhaps 1 Gbps of HTTP protection to keep performing at their best, rather than drop down to ~400 Mbps when doing SSL intercept. Using the ADC’s will keep the firewall running at their top speed and allow all SSL traffic to be inspected.

 

SSL Intercept

Now, when a enterprise client makes a SSL connection, the inside ADC’s will terminate the SSL session by using a local certificate. Yes, you will need to push out that certificate to the trusted store on each PC, via group policy or maybe a login script for Linux machines. The traffic is then passed to the firewalls for cleaning. If the firewalls are happy with the traffic, a new SSL session is created from the outside ADC’s to the real servers. As traffic comes back from the servers, the reverse process happens.

The result is cleaned traffic! Of course, you will also need to have a whitelist that bypasses SSL intercept for certain situations. For example, I am sure people will not want you inspecting their banking transactions.

Links

Filed Under: Blogs, Security Tagged With: , , , , , ,
About Michael Schipp

Michael Schipp - @maschipp on Twitter. An IP professional with 10+ years of enterprise networking experience. Working for A10 Networks in Australia. *My opinions only*

  • Ryan Malayter

    The “big drop in performance” for software SSL is hardware vendor FUD. A single modern x86 core can handle 350+ 2048-bit RSA exchanges per second. You must use HTTP keepalives and TLS session resume anyway for client experience reasons, so that number of SSL handshakes translates to tens of thousands of concurrent users. We run all the traffic for our small SaaS app through two single-core VMs running nginx. And we see 20k concurrent users regularly, but only 0.2 load on the nginx VMs. The IDS/WAF functions are far more CPU intensive than SSL anyway, and cannot be offloaded to specialized hardware.

    • http://twitter.com/maschipp Michael Schipp

      Hi Ryan,
      The drop in performance is stated by the firewall vendors, however the question is how much scale does your company need? If use case is small enough then it would be a good fit to use the software SSL engines in the firewall. However if you are in need of more then you could upgrade/replace the firewall or use ADC’s. The driver here is the move from 2K keys to 4K keys for SSL.

      Thanks
      Michael

      • Ryan Malayter

        Moving from 2k keys to 4k keys would be stupid without also increasing symmetric cipers beyond 256 bits. RSA is not the weak link in the chain at 2k keys. When you already have a million year security margin, adding cost to the most expensive part of a cryptosystem makes zero sense. Unless you’re in marketing.

        I realize that SSL intercept has a different connection and negotiation pattern than hosting SaaS, but a modern 8-core Xeon will still handle browsing for tens of thousands of users with ease. That’s bigger than any corporate campus. No need to write those $100k checks to F5 just to get SSL performance. If load balancing vendors would stop shipping Celeron-class CPUs in their tin they would not need SSL offload at all.

        • http://twitter.com/maschipp Michael Schipp

          From the CPU side, not all vendors use Celeron/AMD, A10 Networks use only Xeon class (no of core goes up with the model). A10 also use software SSL on the SoftAX (Virtual ADC – Though there is a model with hardware SSL called the AXV as well).

          I say again the SSL performance is there for the customer that require it. Sure, some cases will not need that level of performance but some will.

          Note: just adding cores will not give you a linear performance unless your core OS can take advantage of that.

  • Ed Summers

    If I read this correctly, this would basically be client-side proxy of the connection (similar to server-side proxy that would occur with, say, an F5 BigIP LTM that performs SSL offload for a server farm). In that case, would the client receive a warning to the effect that the certificate does not match the domain name being accessed?

    I wouldn’t underestimate the last statement. It of course underscores the need for the user to be aware of the very basics of how the security works. In an Enterprise, where most AUPs have a clause noting that no communications should be considered truly private, this may be less of a concern. However, does it open the Enterprise to potential legal risks should someone’s username/password/account information become compromised?

    • http://twitter.com/maschipp Michael Schipp

      Ed, you are correct the client side will see a certificate warning if the enterprise does not push out the self signed cert via group policy or login scripts, but most enterprise have the controls to place the cert into the trusted store so end user will not see the message.

      Also the legal aspects will very from country to country or maybe even state to state in some country’s.

  • http://marathon-networks.com/ Dan Shechter

    OpenBSD’s relayd can now (in current/snapshots) also do this SSL magic.

    • http://twitter.com/maschipp Michael Schipp

      Thanks Dan, good to know what choices are available.

  • Nik

    Given most enterprises will be using outbound proxy services for HTTPS traffic already, all the major proxy vendors (Blue Coat, Cisco/Ironport, Content Keeper etc) have been doing SSL inspection for many years now.

    It’s easier to do SSL inspection there than pumping money into ADC’s (different story if you already have them around the firewalls, e.g. firewall sandwich) and arguably more secure too – companies like Blue Coat provide assurances that while SSL data is being inspected within the appliance “in-the-clear”, there is no mechanism for an administrator to get that data out of the appliance. That has to be better than renegade administrators with Wireshark pulling peoples sensitive data as it traverses multiple network devices (there would be switches in that topology too I assume?) in-the-clear.

    • http://twitter.com/maschipp Michael Schipp

      Thanks Nik, indeed proxy vendors can be used for SSL Intercept. The main difference is that you only get what that proxy vendor can support (Maleware/AV etc)/detect/clean. Breaking SSL Intercept from the detection engine allows you to chose the best in class engine now and allows for swapping the engine later if something better comes along.

      Your point on security is also valid and worth noting. Switches should not be used in this setup but directly connected with the correct physical security in place. Of course an engineer with a TAP could still sniff the traffic and that is why physical security is needed.

  • NeilTAnderson

    Be _very_ careful with this. Some legal jurisdictions, particularly in the EU, will not tolerate the interception of outbound SSL encrypted traffic, even if it’s part of your acceptable use policy. The intention behind this is to stop inappropriate interception of an employee’s encrypted personal data (such as banking information). Whilst that makes it difficult to inspect outbound SSL traffic for malware, it doesn’t stop you _blocking_ SSL traffic, perhaps based on some sort of reputation (sigh) based list or doing some other form of web filtering.

    It’s also worth bearing in mind that the SSL protocol is designed to detect the (authorised) man in the middle attack that’s going on in the outbound SSL inspection scenario. In this case, you need to ensure that the certificate that is presented by your proxy is trusted by your clients. You may also have to do some hacking with your client web browsers to stop them alerting the user to the non-matching certs that are coming back.

    You can still legitimately intercept inbound traffic of course – especially when using a WAF to protected protected portions of a website. This obviously relies on you having the relevant certs to install on your interception device (WAF or loadbalancer usually).

  • john

    Sure you can intercept SSL traffic. Should you? Never give security people everything they ask for, because for the most part they probably haven’t thought it out. I don’t know why this is, but it’s disturbingly prevalent.

    In this case, if you’ve decrypted ssl traffic, you’re on the hook for it. Someones identity stolen and it’s been tracked down to ssl traffic you’ve intercepted? The FBI never takes “couldn’t have been us” for an answer. And although you get vindicated, you also get investigated. Because you were dumb enough to decrypt ssl traffic.

    So yeah, decrypt away. Just about as smart as asking for facebook passwords.