Support Doc:  Direct Proxy vs Transparent Deployment

Version 2

     

    Introduction

     

    The MWG allows for a multitude of different deployment methods. This article will give an overview of the different options as well as the benefits and the down sides of each of them.

     

    First we need to define a direct proxy and a transparent deployment:

     

    Direct Proxy: The browser/client is "proxy aware" and will actively send traffic to the Web Gateway. A proxy and its options in regards to authentication (as an example) are defined in RFC and commonly supported by browsers and most client apps (like windows media player or java).  MWG offers a direct proxy option as well as a Proxy HA option for redundancy.

     

    Transparent: The browser/client is NOT "proxy aware". The client will try and directly communicate with its destination. The client requests need to be intercepted/redirected on their way to the destination to reach the Web Gateway. As the client is not aware of the Web Gateway or its filtering, all correspondance needs to "look like" it is still happening with the original destination server. Functionality that is normally available to proxies (like authentication) cannot be used in their normal fashion.  MWG offers WCCP, Transparent Router and Transparent Bridge as transparent deployment options.

     

    Often times a transparent deployment is chosen mainly because Administrators do not want to have to make browser changes. Unfortunately this is a common misconception and in reality changes often DO need to be made to browsers (see table below). In addition, transparent deployments often add MORE complexity to the deployment and require more work on the Administration side.

     

     

    Direct vs. Transparent Functionality Comparison

     

    Functionality
    Direct Proxy
    Transparent Deployment
    Traffic DirectionProxy settings need to be pushed to each clientNo settings on the client needed
    Traffic ExceptionsEasily added to Proxy SettingsHas to be done on the network/intercepting device
    Special applications that cannot deal with proxiesWill ignore proxy settings and go directMight fail when getting intercepted transparently
    AuthenticationNative Proxy authentication supported by the browserNeed to use authentication server (redirect client to special authentication URL)
    Promptless authenticationThe browser automatically trusts a proxy with promptless credentialsBrowser needs to be configured to trust the authentication server to send promptless credentials (Browser config on the clients)
    Authentication SessionsNo Sessions needed as each TCP connection is authenticatedSessions (time based or via cookies) are needed. Third party cookies need to explicitly be allowed on each client (Browser config on the clients)
    SSL ScannerNeed to push Root CA to each clientNeed to push Root CA to each client
    SSL TrafficDestination Hostnames are seen by the filtersOnly Destination IPs are seen by the filters
    DNSThe client only needs to talk to MWG. MWG is responsible for DNSThe client and the MWG both need to do their own DNS resolution
    TroubleshootingEasy to rule out the MWG by disabling the proxy on the client temporarilyTo go direct/without the proxy, network changes are required
    TroubleshootingMore control over the traffic flow (which proxy is being used)Harder to determine which proxy is used by a particular client

     

     

    Deploying a Direct Proxy

     

    For a direct proxy deployment, you will need to make the clients/browsers aware of the proxy (it's address and port). here are a few common ways to roll these settings out in your network.

     

     

    Proxy Settings

     

    One way to get the proxy settings out is to manually modify the browser proxy settings.  In most standard browsers you can specify the proxy address and port.  Many browsers also let you specify separate ports for different protocols (FTP, HTTP, HTTPS, etc). Note that for the Web Gateway proxy, you want to select the "same proxy for all protocols" option.

     

    This method is very static and does not give you many options for changes or exceptions.

     

    Proxy.pac/ WPAD.dat

     

    This is one of the more common ways Administrators hand out proxy settings for browsers.  This is a file that contains all of your proxy settings.  It also allows you to create rules, such as bypasses for certain websites.  The proxy.pac file allows for lots of configuration options.  The web gateway also has the ability to host this file with the file server (more in this community article: https://community.mcafee.com/docs/DOC-4916).  There are lots of resources available online if you need help creating your proxy.pac.  A really nice online resource is:  http://findproxyforurl.com

     

    GPO Push

     

    Another very common tool for direct proxy environments.  GPO stands for "Group Policy Object" editor.  In order to use this, you need to be using AD (Active Directory).  This can be used to either push out a proxy pac file to the browsers within your network, or it can be used to enforce and configure the manual proxy settings in each browser within your network.

     

    GPO can also help you with some of the changes needed for transparent deployments (see below).

     

     

     

    Authentication

     

    As seen in the comparison, direct proxy deployments need no or very little considerations for authentication. Transparent deployments on the other hand are much more complex when it comes to authentication.

     

    Below are some considerations when setting up authentication.  For a more detailed look into the different authentication methods and necessary browser changes, please reference this community article: https://community.mcafee.com/docs/DOC-4384

     

     

    Promptless authentication

     

     

    For authentication to be promptless (use windows credentials without prompting the user to enter them) in a transparent environment, you will need to add the Web Gateway IP/hostname into the trusted sites list of your browser.  This has to be done as the Web Gateway redirects you to its internal authentication server, which then asks the browser for credentials. The browser requires an explicit trust to pass the credentials without user interaction.  Failure to trust the Web Gateway IP/hostname will cause the browser to prompt the user to enter in their credentials for every session.

     

     

    Authentication Sessions

     

    In a transparent deployment, authentication needs to be "session based". This means you either have a session bound to your IP address for a certain amount of time (we call this time/IP based) or you need a cookie that is valid for the length of your browser session. This can cause issues in environments where you either have shared workstations (user A's session is still valid when User B starts using the workstation) or where you have multiple users with the same IP (citrix/terminal servers, NAT'ed environments and so on). In these environments you will have to rely on the cookie authentication option.

     

     

    Allowing 3rd Party Cookies

     

    If you are planning on using Cookie Authentication within the Web Gateway, then this would pertain to you.  In many browsers, you will need to enable the security option that allows the browser to receive 3rd party cookies.  Without this enabled you will most likley see malformed and incomplete web pages.

     

     

     

    SSL Traffic

     

    Trusting the Web Gateway Root CA

     

    If you are using the SSL scanner rule set on the Web Gateway, you will need to import the Web Gateway root CA into the browser as a trusted certificate authority.  Otherwise, every time you browse to an SSL site, you will be prompted to trust the certificate. This applies to both, direct and transparent deployments.

     

    More details on the Root CA requirements can be found here: https://community.mcafee.com/docs/DOC-4822

     

     

    Transparent SSL

     

    When a request for an HTTPS site comes in, in a transparent environment, it will come through as an IP address instead of the domain name.  The Web Gateway will then use that IP address as the domain name.  This can cause problems when trying to whitelist\block a domain name or to search for the domain name in an access log.  In order to get the correct domain name instead, we need to add a rule that fixes this.  The rule will essentially take the common name from the certificate, and place that into the url.host property so it is then populated with the correct domain.

     

    For a more detailed explanation about this rule and setup, please reference this community article:  https://community.mcafee.com/docs/DOC-4923

     

     

    DNS

     

    In a transparent environment, DNS is done on the client machine, as it is the client machine that connects to the IP address of the web server.  When the request makes it to the Web Gateway, the Web Gateway will also do a DNS request before making its request out to the web server.  In some situations this can lead to complications, for example when you get different results from the DNS server the client uses and from the DNS server the Web Gatewayuses. It can get even worse if the client is able to resolve a particular hostname via DNS, but the MWG is not able to. It is strongly recommended that you make sure that your clients and your Web Gateway use the same DNS server(s).

     

    Troubleshooting Considerations

     

    Below are some items to take into consideration when troubleshooting issues that may arise in each deployment method.

     

    Request Line in TCP Dumps

     

    Often times when troubleshooting, you might need to look at a TCP Dump and filter for HTTP requests.  When looking at a TCP Dump/Packet Capture you will notice differences in how the GET request looks.  In both examples below, I have filtered for http.request.

     

     

      • In a Direct Proxy Deployment you will see the absolute URL in the request line, which is much easier to follow, as shown in this screenshot:

     

     

    GET-Direct-Proxy.png

     

     

     

      • In a Transparent Deployment, you will see the relative URL in the request line, which is more difficult to follow.  So the Host header will show the domain, and the request line will only show the relative path as shown in this screenshot:

     

     

    GET-Transparent.png

     

     

     

    SSL Connection Differences

     

    In a transparent environment, HTTPS requests will come through as an IP address instead of a domain name.  This can make things tricky when trying to filter through a TCP Dump for a request, or even when looking at an access log.   In a direct proxy setup this is not the case as there is a CONNECT request first which contains the domain name.

     

     

      • Here is an example of an HTTPS request in a TCP Dump using a Direct Proxy Deployment, where the CONNECT request is clearly seen and easy to follow:

     

     

    Proxy-SSL.png

     

     

     

      • Here is an example of an HTTPS request in a TCP Dump using a Transparent Deployment.  Here it is much more difficult to see as there is no CONNECT request, and you can see the client is connecting to the actual IP of the website:

     

     

    WCCP-SSL.png

     

     

     

    Following a client request in a TCP Dump

     

    In a Direct Proxy Deployment, you can very easily see the client IP connect to the web gateway IP over the proxy port, and it is very easy to troubleshoot.  See example:

     

     

    Proxy-TCP.png

     

     

     

    However, in a Transparent Deployment, it can be tricky trying to trace a request, from a client, in a TCP Dump.  The client will connect to the actual IP of the web server, instead of the Web Gateway IP.  Then, the Web Gateway will make a request to the web server IP on behalf of the client.  Finally, when responding back to the client, the Web Gateway will spoof the web server IP.  This can be confusing when trying to trace traffic in a capture.  See example:

     

     

    WCCP-TCP.png

     

    Best Practice: Support Recommendations

     

    Now that we have discussed the different deployment options and considerations, let us take a minute and talk about best practices from a support perspective. As you can imagine, technical support has seen every deployment scenario imaginable over the years.

     

    Before we start, lets be clear that all of the deployment methods mentioned here are fully supported and function as designed. The below should be taken purely as supports experience with the different deployment methods.

     

    MWG in its core is a proxy product. Therefore the hands down most successful deployment method out there is a combination of Direct Proxy with proxy.pac and a hardware load balancer in front of the Web Gateways. This deployment gives you enterprise wide flexibility when it comes to change management (easy changes in proxy.pac once deployed), high availability and load balancing as well as maintenance and troubleshooting.

     

    If you do decide to go with a transparent deployment method, the best experience we have is with WCCP. WCCP as a protocoal has many enterprise features already build in and we often see it in addition to the direct proxy deployments. For example to cover a guest wireless network or to have a fall back/catch all for devices not covered by the direct proxy settings. More on WCCP here: https://community.mcafee.com/docs/DOC-4917

     

    If you do decide to deploy either transparent router or transparent bridge modes, the most common issue we see in support is with exceptions. Everyone needs exceptions at some point (internal resources, non standard compliant apps and so on). In transparent router mode you can "route" around the Web Gateway on a logical level. In transparent bridge mode, the Web Gateway is in the physical path of the traffic (inline) and it is almost impossible to get around it. Also important for bridge mode is the potential for a complete network outage if no precautions are taken before a Web Gateway goes offline.