Showing results for 
Show  only  | Search instead for 
Did you mean: 

Web Gateway: Considerations when Whitelisting/Blacklisting HTTPS URLs


This document will cover the configuration of how to whitelist or blacklist SSL (HTTPS) based websites while using the SSL scanner.  There are many different configurations which a ruleset can be configured with however so we will provide some best practice ideas when implementing rules along with some information into how the SSL (HTTPS) process works.


In order to block or allow SSL traffic on the Web Gateway, you are going to need to have some basic knowledge of how SSL traffic works with proxies and what needs to be done to inspect SSL traffic.  One major thing to take into consideration is that SSL traffic is different from HTTP traffic as more steps are needed to ensure the encryption of the traffic.

In HTTP, the full URL ( is transmitted in clear text and whitelisting (or blocking) can apply to the full URL.

For HTTPS requests, only the Host ( is transmitted in clear text. The full URL is encrypted inside the SSL Tunnel. To whitelist the full URL SSL inspection is needed and you have to ensure that the request is not blocked, before the full URL has been transmitted in the tunnel.

Here is an overview of the difference between the two:



This is what a HTTP connection looks like in Wireshark:



This is what happens during a HTTPs (SSL) “CONNECT” command;



Here is an example of a “CONNECT” attempt from the client which contains the HTTP header information and only the “Host”, not the full URL.  This is what the Web Gateway will be able to see if the request is “tunneled” or the “SSL Scanner” is not used.



 SSL scanning can be much more granularly controlled in the current versions (above 7.7.x) by using rule criteria associated with the SSL     scanning ruleset.

 For example, certificate check or content check bypasses can be based on any combination of criteria including certificates, user agent,       destination IP, destination category, etc.

 There are 2 options on how MWG can use a client certificate:

  1. It is possible that MWG uses a client certificate in the https connection to the server.
  1. This above statement stands true only if the connection will not be tunneled. It will also work if the client connection is HTTP and only the connection to the server is https.
  2. It selects a predefined certificate, independent from the client.

     2. MWG will not send the client certificate in advance. It will send it only if it is configured to do so and if the server asks for it.

There are two different stages when a server may ask for a client certificate:

First during the initial SSL handshake

Second during the re-negotiation sequence.

The HTTPs connection goes through two cycles- the first being the "CONNECT CALL" and the second cycle being the "CERTVERIFY" cycle.

Therefore, if the HTTPs CONNECT call is allowed through, we should be passing the certs to fully establish the SSL communications between endpoints of the two tunnels "Client ---Tunnel1---> Web Gateway---Tunnel2---> Web Host".

After the Web Gateway gets the SSL Certificate, it will then process the certificate validation rules in the SSL Scanner ruleset against the server cert to determine if the connection should be allowed through to the site.

Two options:

  1. to use a client certificate list (1) or
  2. to use a predefined certificate (2)





Both modes allow the customer to import certificates together with an RSA or DSA private key with a minimum size of 1024 bit. In a FIPS environment, the minimal size is 2048 bit. A certificate chain can/needs only to be imported for a predefined client certificate because in the other case the certificate chain sent by the client will be forwarded to the server.

When working with SSL traffic, there are two very important scenarios to consider:


1. When working without SSL scanner you will need to keep the following things in mind;

      • After the “CONNECT” occurs, the data will be encrypted.
      • The “CONNECT” only contains the destination host and not the full URL (and very few other headers that can be read).
      • The Steps below do NOT apply to setups without SSL Scanner. Without SSL Scanner, only hosts, not full URLs can be whitelisted


2. When working with SSL scanner you will need to keep the following things in mind;

      • The “CONNECT” only contains the destination host and not the full URL (and very few other headers that can be read).
      • If the CONNECT is blocked, the full URL will never be visible to the web gateway
      • If the CONNECT is NOT blocked, the traffic will be decrypted on the Web Gateway and at this time, the full url will visible.
      • When using the SSL Scanner, there are going to be two different “Request” Cycles that will need to occur;
        • One for equals CONNECT
        • One for equals CERTVERIFY

(Both of these “Request Cycle” events need to be allowed for the full URL to be passed in the SSL Tunnel in order to allow the whitelisting to occur.)

Example Scenario

In our case, the website “” is blocked based on the “Information Security” category but still users should be allowed to go to one specific URL on the site:

If this were to be an HTTP request you would block the category “Information Security” in your URL Filter block rules and then allowing this one specific URL in an exception before the blocking rule (for example, the global whitelist).  This will not work with SSL/HTTPS traffic;




Blocking Rule



For HTTPS requests this results in the following. The CONNECT request is being blocked:



The Issue is, that the CONNECT request the client made for the https site, did NOT contain the full URL, but only the host As a result, your whitelist did not match, and the category block was applied.

To resolve this issue, you would need to make a rule to allow the “CONNECT” command and “CERTVERIFY” to pass through without getting blocked.  We know what the URL.HOST is so you could make a rule like the following;


At this point it is very important to check your brackets. You can add brackets during the rule creation:


This will then allow the “CONNECT” part of the SSL traffic to pass through the Web Gateway allowing the tunnel to be established so that the full URL can be requested in the tunnel.

Note that no actual traffic (requests or responses) are exchanged during the CONNECT. Only the connections are established and certificates are exchanged. The “GET” request inside the tunnel, is still being filtered as normal with these new rules.

The Web Gateway will now be able to decrypt the traffic and we will be left with the following as seen in a connection trace using the SSL Scanner;

Decrypted Traffic.jpg


In conclusion, the site will now load correctly for the part of the site which we did not want to have blocked (Full URL added to your whitelist earlier) while yet blocking the rest of the site.  Granted, it might be missing a style sheet and some pictures but this is because we only allowed the one very specific URL. The other URLs could be added to the whitelist as needed.

Please keep in mind that when trying to troubleshoot SSL traffic, you will have to use Connection Tracing to see inside the tunnel.  This is enabled in the “Configuration > Troubleshooting” section and the output will be in the “Troubleshooting > Connection Tracing” section.

Additionally, if you plan to use the connection tracing for troubleshooting, please enable it but restrict it to one test client or you can risk filling up the Web Gateway.

Furthermore, the client-side of the tunnel will be the traces labeled like “-C.txt” and the Web Server side of the tunnel will be labeled like “-S.txt”.


With this information, you should now be able to whitelist and blacklist URLs on the Web Gateway.  Furthermore, you should have some additional knowledge into some of the troubleshooting steps which can be implemented for troubleshooting SSL (HTTPs) traffic as well.

Labels (1)

Excellent article.  Still a bit confused though.  Wouldn't we want to have this as the norm for SSL for all sites?  Why limit it to just a site you are trying to block but allow a specific deep link?



Very simple! Allow users to access "" and "" but block Social Media Category. In this case you need exactly what is explained above.




Hello Jspanitz,

For this Best Practices article, the goal was to show that if there is a block occurring on the initial "CONNECT" to the host, you would not have the ability to allow a specific long URL as this would be passed in the SSL tunnel after the CONNECT phase has been completed.  If you want to have the Web Gateway block specifically on the URL.Host for the site, you could just let the traffic for the CONNECT to proceed down the ruleset where it would be subject to the filtering rules.  However, if this behavior is not desired, or you would like to process the filtering on the full URL path, you would need to do a stop cycle on the initial connect phase so that this traffic is not subjected to the filtering later in the ruleset.  In the end, this could be more of a preference-based configuration as some customer's will want to block the URL.Host outright.

In the end, it is up to your discretion for what behavior you would like to see.


Nice Doc.


I made rulesets as your guide but it didn't work for my test.

From the above guide, you didn't mention about "SSL Scanner" ruleset, Does this rulese affected to this configuration ? And should we enable "SSL Scanning" for this website or this category ?

Thank you

Version history
Revision #:
5 of 5
Last update:
‎12-17-2020 07:28 AM
Updated by:

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community