5 Replies Latest reply on Dec 17, 2010 11:16 AM by eelsasser

    overload detected : memory usage - send http 503

      Hi,

       

      I am running McAfee Web Gateway 6.8.7 build 8846

       

      The main problem I am having is when contant that should be blocked, is in fact blocked, but the user does not get a blocked message page. They just get a blank page.

       

      Here is the last several lines from the feedback file errors.log:

       

       

       

      [16/Dec/2010:13:55:40 -0600] overload detected: memory usage 2784067584
      [16/Dec/2010:13:56:25 -0600] overload timeout: memory usage still 3159760896, action is accept and send http 503
      [16/Dec/2010:14:04:22 -0600] end of overload: memory usage 1026248704
      [16/Dec/2010:14:04:38 -0600] overload detected: memory usage 2623152128
      [16/Dec/2010:14:05:23 -0600] overload timeout: memory usage still 2629468160, action is accept and send http 503
      [16/Dec/2010:14:08:25 -0600] end of overload: memory usage 1753550848
      [16/Dec/2010:14:09:41 -0600] overload protection has been activated once
      [16/Dec/2010:14:24:17 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:24:17 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:24:24 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:25:37 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:25:37 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:26:30 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:26:30 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:26:39 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:26:49 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:27:42 -0600] Central Management: failed to send request 'https://x.x.x.x:10000/exec?command=syncCluster&clientCmd=UpdateQuota&deltaVersio n=362' while doing general operation:
      [16/Dec/2010:14:27:42 -0600] Quota sync: Failed to sync quota with the master with code 0
      [16/Dec/2010:14:27:51 -0600] Central Management: failed to send request 'https://x.x.x.x:10000/exec?command=syncCluster&clientCmd=getClusterPassword' while doing general operation:
      [16/Dec/2010:14:39:21 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:39:21 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:39:46 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:39:46 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:39:50 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:39:53 -0600] Failed to load host key '/opt/webwasher-csm/conf/ssh/ssh_host_rsa_key'
      [16/Dec/2010:14:48:24 -0600] WebUI: External redirect URL 'https://172.19.96.14:10000/conf?navTo=Config&foo=48140000&userID=Default-Faceboo k&session=Kq91QVf/PHrdxVo' ignored
      [16/Dec/2010:15:07:09 -0600] end of overload: memory usage 1914458112
      [16/Dec/2010:15:07:27 -0600] overload detected: memory usage 2623959040
      [16/Dec/2010:15:08:12 -0600] overload timeout: memory usage still 2629222400, action is accept and send http 503
      [16/Dec/2010:15:18:09 -0600] Central Management: failed to send request 'https://x.x.x.x:10000/exec?command=syncCluster&clientCmd=GetUpdate&type=AV&versi on=7111.70.2743&engine=8&ossubtype=cglinux-5.0-intel' while doing general operation: Cannot allocate memory
      [16/Dec/2010:15:18:10 -0600] Update verification from Master failed.

       


      [16/Dec/2010:15:18:41 -0600] McAfee micro incremental update failed: 1
      [16/Dec/2010:15:18:41 -0600] McAfee Gateway Anti-Malware Engine: failed to load MFE base API in '/opt/webwasher-csm/conf/antivirus/SCANM7111.72.2743'
      [16/Dec/2010:15:18:41 -0600] Failed to load and initialize McAfee Gateway Anti-Malware engine, error code 3
      [16/Dec/2010:15:20:46 -0600] end of overload: memory usage 1809698816
      [16/Dec/2010:15:21:03 -0600] overload detected: memory usage 2623414272
      [16/Dec/2010:15:21:48 -0600] overload timeout: memory usage still 2656141312, action is accept and send http 503

       

       

       

       

      When a perform a tcpdump on the web gateway interface I see the http 503 responses going to the client. Anyone Have an Idea have to correct the http 503 errors and get web gateway to 'display' a meaning blocked message page? How to correct the memory overload errors?

       

       

      Thanks in advance.

       

       

      Message was edited by: gsxr1000 on 12/16/10 3:44:00 PM CST

       

       

      Message was edited by: gsxr1000 on 12/16/10 3:45:33 PM CST
        • 1. Re: overload detected : memory usage - send http 503
          Jon Scholten

          Hey gsxr1000!


          Overload errors typically stem from the Web Gateway failing to be able to allocate enough memory to proceed with filtering requests.


          Most situations this will be alleviated with a restart of the service from the CLI (service webwasher-csm restart). It can be mitigated by enabling Maintenance options under Configuration > Maintenance, then enforcing it (so it actually takes place). If it is a true overload and there are tooooo many reuqests, then it will most likley occur regardless of Maintenance or a restart of the service.


          If you look around the community you will see people talking about MP or Multi-Process mode. This mode essentially tripples or septupples the processing power of the appliance, enabling it to handle more requests. For more information on the multi-process mode you can read more about it in the System Configuration Guide, it can be found under Home > Manuals in the Web Gateway's (Webwasher) interface, info on MP-mode is located on page 155. Alternativley it can be found in our KB at https://kc.mcafee.com/corporate/index?page=content&id=PD22641.


          SO, if you have a 1900, 2900, 5000, or 5550 model appliance, I would strongly recommend enabling MP mode. Otherwise if you are seeing overloads occur on a smaller model quickly after a restart the appliance may be truley overloaded.


          ~Jon

          • 2. Re: overload detected : memory usage - send http 503

            Hi Jon,

             

            Thank you for the info, I will read about MP. This appliance is not in production Yet and virtually has no load on it ( see attachment ). We are just trying to test our policies to make sure they work correctly and can not get any blocked error pages to display in the browser. the appliance just returns a http 503 saying the service is unavailable and the error log displays memory overload errors. This is a WG-5000 with. Any other thoughts?

             

            Thank you.

            • 3. Re: overload detected : memory usage - send http 503

              Despite the low traffic load, I've seen some instances where network conditions can start looping traffic into itself.

               

              For example, if you have WCCP turned on to intercept port 80 AND a proxy listener on port 80 directly instead of a redirect to the port 9090 AND Via: headers are turned off AND certain auto-updating software that sends malformed requests, and you have asymmetrical routes, AND you still have another filtering solution inline (like and IPS) that terminates connections, AND it's a full-moon.

               

              Not saying that you have this problem, just that it may not really be load related.

              • 4. Re: overload detected : memory usage - send http 503

                e²  Traffic that is not blocked by policies routes OK. WCCP is turned off. I have attached the listening ports. The client and the WG both display a http 503 entry in their tcpdump files. I don't think that equates to a routing issue. It is not a full moon till Dec 21st. Thoughts?  Thank you.

                • 5. Re: overload detected : memory usage - send http 503

                  The only other thing that comes to mind is if you are using an explicit proxy, do you have the internal IP 172.x.x.x address space excluded in the broiwser from being proxied?

                   

                  The error.log you showed also had an AV update failure right before, which is suspicious to me, but don't know if it's related.

                   

                  You might have to call support.