cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Considerations when Whitelisting/Blacklisting HTTPS URLs

Web Gateway: Considerations when Whitelisting/Blacklisting HTTPS URLs

Introduction

This document will cover the configuration of how to whitelist or blacklist SSL (HTTPS) based websites while using the SSL scanner.  There are many different configurations which a ruleset can be configured with however so we will provide some best practice ideas when implementing rules along with some information into how the SSL (HTTPS) process works.

Overview

In order to block or allow SSL traffic on the Web Gateway, you are going to need to have some basic knowledge of how SSL traffic works with proxies and what needs to be done to inspect SSL traffic.  One major thing to take into consideration is that SSL traffic is different from HTTP traffic as more steps are needed to ensure the encryption of the traffic.

In HTTP, the full URL (www.domain.com/images/logo.gif) is transmitted in clear text and whitelisting (or blocking) can apply to the full URL.

For HTTPS requests, only the Host (www.domain.com) is transmitted in clear text. The full URL is encrypted inside the SSL Tunnel. To whitelist the full URL SSL inspection is needed and you have to ensure that the request is not blocked, before the full URL has been transmitted in the tunnel.

Here is an overview of the difference between the two:

Diagram1.jpg

This is what a HTTP connection looks like in Wireshark:

image3.png

This is what happens during a HTTPs (SSL) “CONNECT” command;

image4.png

Here is an example of a “CONNECT” attempt from the client which contains the HTTP header information and only the “Host”, not the full URL.  This is what the Web Gateway will be able to see if the request is “tunneled” or the “SSL Scanner” is not used.

image5.png

When working with SSL traffic, there are two very important scenarios to consider:

1. When working without SSL scanner you will need to keep the following things in mind;

      • After the “CONNECT” occurs, the data will be encrypted.
      • The “CONNECT” only contains destination host and not the full URL (and very few other headers that can be read).
      • The Steps below do NOT apply to setups without SSL Scanner. Without SSL Scanner, only hosts, not full URLs can be whitelisted

2. When working with SSL scanner you will need to keep the following things in mind;

      • The “CONNECT” only contains destination host and not the full URL (and very few other headers that can be read).
      • If the CONNECT is blocked, the full URL will never be visible to the web gateway
      • If the CONNECT is NOT blocked, the traffic will be decrypted on the Web Gateway and at this time, the full url will visible.
      • When using the SSL Scanner, there are going to be two different “Request” Cycles that will need to occur;
        • One for command.name equals CONNECT
        • One for command.name equals CERTVERIFY

(Both of these “Request Cycle” events need to be allowed for the full URL to be passed in the SSL Tunnel in order to allow the whitelisting to occur.)


Example Scenario

In our case, the website “csm-testcenter.org” is blocked based on the “Information Security” category but still users should be allowed to go to one specific URL on the site:

https://www.csm-testcenter.org/test?do=show&subdo=client_information

If this were to be an HTTP request you would block the category “Information Security” in your URL Filter block rules and then allowing this one specific URL in an exception before the blocking rule (for example, the global whitelist).  This will not work with SSL/HTTPS traffic;

Whitelist:

image7.png

Blocking Rule

image6.png

For HTTPS requests this results in the following. The CONNECT request is being blocked:

image8.png

The Issue is, that the CONNECT request the client made for the https site, did NOT contain the full URL, but only the host www.csm-testcenter.org. As a result, your whitelist did not match, and the category block was applied.

To resolve this issue, you would need to make a rule to allow the “CONNECT” command and “CERTVERIFY” to pass through without getting blocked.  We know what the URL.HOST is so you could make a rule like the following;

image9.png

This will then allow the “CONNECT” part of the SSL traffic to pass through the Web Gateway allowing the tunnel to be established so that the full URL can be requested in the tunnel.

Note that no actual traffic (requests or responses) are exchanged during the CONNECT. Only the connections are established and certificates are exchanged. The “GET” request inside the tunnel, is still being filtered as normal with these new rules.

The Web Gateway will now be able to decrypt the traffic and we will be left with the following as seen in a connection trace using the SSL Scanner;

Decrypted Traffic.jpg

In conclusion, the site will now load correctly for the part of the site which we did not want to have blocked (Full URL added to your whitelist earlier) while yet blocking the rest of the site.  Granted, it might be missing a style sheet and some pictures but this is because we only allowed the one very specific URL. The other URLs could be added to the whitelist as needed.

Please keep in mind that when trying to troubleshoot SSL traffic, you will have to use Connection Tracing to see inside the tunnel.  This is enabled in the “Configuration > Troubleshooting” section and the output will be in the “Troubleshooting > Connection Tracing” section.

Additionally, if you plan to use the connection tracing for troubleshooting, please enable it but restrict it to one test client or you can risk filling up the Web Gateway.

Furthermore, the client side of the tunnel will be the traces labeled like “-C.txt” and the Web Server side of the tunnel will be labeled like “-S.txt”.

Conclusion

With this information, you should now be able to whitelist and blacklist URLs on the Web Gateway.  Furthermore, you should have some additional knowledge into some of the troubleshooting steps which can be implemented for troubleshooting SSL (HTTPs) traffic as well.

Comments
jspanitz

Excellent article.  Still a bit confused though.  Wouldn't we want to have this as the norm for SSL for all sites?  Why limit it to just a site you are trying to block but allow a specific deep link?

asabban

Hi!

Very simple! Allow users to access "www.facebook.com/yourcompany" and "www.twitter.com/yourcompany" but block Social Media Category. In this case you need exactly what is explained above.

Best,

Andre

abenjami

Hello Jspanitz,

For this Best Practices article, the goal was to show that if there is a block occurring on the initial "CONNECT" to the host, you would not have the ability to allow a specific long URL as this would be passed in the SSL tunnel after the CONNECT phase has been completed.  If you want to have the Web Gateway block specifically on the URL.Host for the site, you could just let the traffic for the CONNECT to proceed down the ruleset where it would be subject to the filtering rules.  However, if this behavior is not desired, or you would like to process the filtering on the full URL path, you would need to do a stop cycle on the initial connect phase so that this traffic is not subjected to the filtering later in the ruleset.  In the end, this could be more of a preference-based configuration as some customer's will want to block the URL.Host outright.

In the end, it is up to your discretion for what behavior you would like to see.

m.bagheryan

Nice Doc.

dohi1

I made rulesets as your guide but it didn't work for my test.

From the above guide, you didn't mention about "SSL Scanner" ruleset, Does this rulese affected to this configuration ? And should we enable "SSL Scanning" for this website or this category ?

Thank you

Version history
Revision #:
1 of 1
Last update:
‎03-22-2013 10:35 AM
Updated by: