6 Replies Latest reply on Aug 17, 2016 1:37 PM by Jon Scholten

    MCP: How to improve speed in switching between proxy servers

    malware-alerts

      Using MCP with "connect to proxy server with fastest response time".


      2 defined proxies: our internal MWG and the McAfeeSaaS proxy.


      The internal MWG only responds when clients are on corporate LAN.

      The McAfeeSaaS proxy only responds when clients are on an external network that has access to the internet (we block proxy traffic to the McAfeeSaaS service on our corporate LAN)


      When clients switch between networks (internal to external or vice-versa), there is a fairly significant delay before MCP switches to a responding proxy.


      Did some test and took a few TCP traces on a few clients and typically this is what we see:


      1. User switches network connection.
      2. Immediately upon a new IP being assigned, MCP will send a SYN request to the previously used proxy (no answer)
      3. 3 seconds later, MCP will retransmit a SYN request to the previously used proxy (no answer)
      4. 6 seconds later, MCP will retransmit a SYN request to the previously used proxy (no answer)
      5. ~40 seconds later, MCP will send a SYN request to the first proxy in the MCP policy list.
      6. First proxy in list responds (Full SYN, SYN/ACK, ACK, FIN/ACK, FIN/ACK, ACK sequence).
      7. MCP sends a new SYN request to second proxy in list, the one tat was previously used (no answer)
      8. 3 seconds later, MCP retransmit a SYN request to the second proxy in list (no answer)
      9. 6 seconds later, MCP retransmit a SYN request to the second proxy in list (no answer)
      10. ~80 seconds later, MCP will send a SYN request to the first proxy in the MCP policy list.
      11. First proxy in list responds (Full SYN, SYN/ACK, ACK, FIN/ACK, FIN/ACK, ACK sequence).
      12. User is able to browse. (total ET: ~140 seconds after IP was changed)

       

      My questions:

       

      1. Has anybody else noticed this delay?
      2. From what I'm assuming in the TCPdump behavior from the clients, when using "connect to proxy server with fastest response time", MCP will systematically try both proxy servers in it's list before taking a decision on which one is fastest, even if one of them answers (complete SYN-SYN/ACK-ACK sequence) before the SYN request to the other one has timed-out.
        • If my assumption is right, the behavior is not very efficient (MCP should immediately assume the first one that answers is the fastest instead of waiting for both to answer), is there anyway to change the behavior?

       

       

      As a side note, I've also tested with "connect to the first accessible proxy server based on their order in the list below" and it does shave some time in selecting the available proxy when switching between networks for as long as the first proxy in the list is the one that is reachable on the new connection (looking at ~40 seconds delay). If the second in list is the one that answers, the overall delay is roughly around ~60-70 seconds.

        • 1. Re: MCP: How to improve speed in switching between proxy servers
          Jon Scholten

          Hi malware-alerts!

           

          Your observations are correct, though that does seem a bit longer from what I've seen. This could vary I suppose based on what action the firewall may take (drop or reset -- this isnt always controllable..).

           

          At the moment, MCP 1.2 checks each of the proxy servers in order one-by-one. This has actually changed in MCP 2.0 (for Mac only right now), where MCP performs all the checks in parallel (each proxy, captive portal, etc...). The result with the parallel check is faster proxy selection. There are plans to change the Windows version to this method, not sure of a timeline just yet though.

           

          I would suggest sticking with "order preference" instead of "fastest proxy" especially when using MWG on premise.

           

          If you have a ticket open please let me know it and I will very gladly bring this up with PM and dev to make sure it has their attention.

           

          Best Regards,

          Jon

          • 2. Re: Re: MCP: How to improve speed in switching between proxy servers
            malware-alerts

            Jon,

             

            Just as an FYI, I do have a case opened for the delay, I'll wait for the engineer to complete his analysis, but from what I can gather in the MCP verbose logs, the major part of the delay happens when MCP tries to test for the captive portal (mcp.webwsaher.com). If this is blocked at the firewall, it causes a 60+ seconds delay before trafic is finally redirected.

             

            In my case, the captive portal request is blocked at the FW. The FW doesn't answer with a RST and lets the connection timeout (drop). From what I can see in packet captures, a First SYN is sent, then a second SYN is sent 3 seconds later, and a third SYN is sent 6 seconds later. Assuming the third SYN times-out 12 seconds after it is sent, this would account for 21 seconds overall from the first SYN (3 seconds + 6 seconds + 12 seconds). For some reason, if you look at Step5 in the example below, the delay is actually of 42 seconds on the first captive portal test attempt and then another 21 seconds on the secondary attempt.

             

            MCP.LOG example::

             

            Step1: Client's IP Address change happens here: (the new IP address gets assigned @ 14:17:09)

            07/20/15 : 14:17:05:302 - [VERBOSE] CNetworkManager::run -  Detected route change

            07/20/15 : 14:17:05:302 - [VERBOSE] CNetworkManager::checkConnectivity -  Checking connectivity status

            07/20/15 : 14:17:05:302 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  Checking proxy connectivity

            07/20/15 : 14:17:05:302 - [WARNING] CNetworkManager::checkTCPConnect -  Cannot connect to address [internalproxyIPaddress] and port [internalport]

            07/20/15 : 14:17:05:302 - [WARNING] CNetworkManager::checkTCPConnect -  Cannot connect to address [mcafeeSaaS_IPaddress] and port [8080]

            07/20/15 : 14:17:05:318 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  No proxy connectivity

            07/20/15 : 14:17:05:318 - [VERBOSE] CNetworkManager::checkConnectivity -  Always redirecting, skipping corporate connectivity tests

            07/20/15 : 14:17:05:380 - [VERBOSE] CNetworkManager::run -  Detected adapter address change

            07/20/15 : 14:17:05:380 - [VERBOSE] CNetworkManager::checkConnectivity -  Checking connectivity status


            Step2: MCP detects proxy connectivity (first proxy in the policy list): (This can be seen as a full SYN, SYN/ACK, ACK, FIN/ACK, FIN/ACK, ACK in a packet capture)

            07/20/15 : 14:17:05:396 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  Checking proxy connectivity

            07/20/15 : 14:17:14:413 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  Detected proxy connection at 9017ms

             

            Step3: MCP also sends a SYN to [mcafeeSaaS_IPaddress] @ 14:17:14:417 in packet capture and times-out 11 seconds later:

            07/20/15 : 14:17:35:432 - [WARNING] CNetworkManager::checkTCPConnect -  Cannot connect to address [mcafeeSaaS_IPaddress] and port [8080]

             

            Step4: MCP does a DNS query and test connectivity to the captive portal:

            07/20/15 : 14:17:35:432 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  Using proxy policy order

            07/20/15 : 14:17:35:432 - [VERBOSE] CNetworkManager::checkConnectivity -  Always redirecting, skipping corporate connectivity tests

            07/20/15 : 14:17:35:432 - [VERBOSE] CNetworkManager::checkCaptivePortalConnectivity -  Testing captive portal

            07/20/15 : 14:17:35:432 - [VERBOSE] CNetworkManager::getUrlWebPage -  Getting url

             

            Step5: Timeout on the first captive portal test (42 seconds after first attempt):

            07/20/15 : 14:18:17:472 - [ERROR] CNetworkManager::getUrlWebPage -  Failed sending request 12002

            07/20/15 : 14:18:17:472 - [ERROR] CNetworkManager::getUrlWebPage -  Connection rejected or timed out, behind firewall

            07/20/15 : 14:18:17:472 - [VERBOSE] CNetworkManager::checkCaptivePortalMethod1 -  Failed getting web page

             

            Step6: Timeout on the second captive portal test (21 seconds after second attempt):

            07/20/15 : 14:18:17:472 - [VERBOSE] CNetworkManager::getUrlWebPage -  Getting url

            07/20/15 : 14:18:38:554 - [ERROR] CNetworkManager::getUrlWebPage -  Failed sending request 12002

            07/20/15 : 14:18:38:554 - [ERROR] CNetworkManager::getUrlWebPage -  Connection rejected or timed out, behind firewall

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::checkCaptivePortalMethod2 -  Failed getting web page

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::checkCaptivePortalConnectivity -  All captive portal connection attempts rejected, firewall detected

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::setConnectionState -  Setting new connection state 2


            Step7: Finally MCP redirects traffic to available proxy (93 seconds after route change was detected)

            07/20/15 : 14:18:38:554 - [VERBOSE] CRulesEngine::run -  Received network detection request

            07/20/15 : 14:18:38:554 - [VERBOSE] CRulesEngine::run -  Enforcing redirection after network change

            07/20/15 : 14:18:38:554 - [VERBOSE] CFireCorePlugin::CreateForwardingRules -  Creating policies & rules

            07/20/15 : 14:18:38:554 - [VERBOSE] CFireCorePlugin::CreateForwardingRules -    Adding Match for port 80

            07/20/15 : 14:18:38:554 - [VERBOSE] CFireCorePlugin::CreateForwardingRules -    Adding Match for port 443

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::run -  Initiating periodic network connectivity check when no connectivity detected

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::checkConnectivity -  Checking connectivity status

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  Checking proxy connectivity

            07/20/15 : 14:18:38:554 - [VERBOSE] CNetworkManager::checkProxyConnectivity -  Detected proxy connection at 0ms

            07/20/15 : 14:18:39:177 - [VERBOSE] CFireCorePlugin::CStreamPlugin::RedirectOutboundConnect -  Received Firecore redirected data from Application : C:\PROGRAM FILES\INTERNET EXPLORER\IEXPLORE.EXE

             

            My questions:

            • Why is MCP bothering with checking the second proxy in my list in Step3 if it already detected a connection to the first proxy in Step2 ?
            • What's with the 42 seconds timeout delay in Step4 (Captive portal check) when the actual TCP timeout should have been around 21 seconds?
            • 3. Re: Re: MCP: How to improve speed in switching between proxy servers
              Jon Scholten

              HI MA!

               

              Thank you very much for this detailed analysis. I will bring it up with development.

               

              On #1, if you have "use fastest proxy" this makes sense because we're measuring the response time for proxy 1 and proxy 2.

               

              On #2, I'm not sure why MCP would wait that long.

               

              Can you let me know the SR #?

               

              Best Regards,

              Jon

              • 4. Re: Re: MCP: How to improve speed in switching between proxy servers
                malware-alerts

                Jon Scholten wrote:

                On #1, if you have "use fastest proxy" this makes sense because we're measuring the response time for proxy 1 and proxy 2.

                 

                I'm now actually running with "order preference" as you recommended.

                 

                Jon Scholten wrote:

                Can you let me know the SR #?

                Sent you a PM.

                 

                Thanks.

                • 5. Re: MCP: How to improve speed in switching between proxy servers
                  malware-alerts

                  Looks like I can't send you a message since we're not connections.

                  • 6. Re: MCP: How to improve speed in switching between proxy servers
                    Jon Scholten

                    Hi malware-alerts,

                     

                    Just an FYI, MCP 2.2 should now have faster connection checks, this can be a cause for delay in making the proxy active. It was released on August 4th.

                     

                    I started posting MCP release notes with MWG release notes here: McAfee Web Gateway Release Notes

                     

                    In the MCP release notes its listed as:

                    -Enhanced proxy polling logic with simultaneous polling of all configured proxy servers

                     

                    In your case you were switching networks, so I am not 100% sure it cuts it down all the way, but should cut down on proxy check time.

                     

                    Best Regards,
                    Jon