1 2 3 Previous Next 73 Replies Latest reply on Feb 25, 2016 10:57 PM by norbertg

    High memory usage on servers since upgrading to VSE 8.8 P6

    steveholleyuk

      We have been having big issues with apparent memory leaks on our Windows Hyper-V 2008 and 2012 hosts, since upgrading VirusScan 8.8 to Patch 6. The processes are mferkdet.sys and mfehidk.sys.


      The servers are grinding to a halt with these processes taking all the available RAM :-


      Anyone else seen this?

       

       

        • 1. Re: High memory usage on servers since upgrading to VSE 8.8 P6
          wwarren

          I haven't.

          Forcing a dump of the system while in that state would be helpful. Plus running our MER tool afterwards so we can see what else is installed, configurations, etc.

           

          mfehidk.sys is likely blameless; it'll show up in everything though because its purpose in life is a kin to a telephone operator, funneling information off to the appropriate interested party.

          mferkdet.sys is a root-kit detection driver. So you'll see it do most its work when your on-demand scan task runs, where the task is configured to scan memory for rootkits. It also suggests you can control the rate of the leak by controlling your ODS tasks.

          It's plausible there's an interop issue with another driver on the system that is causing us to leak memory. We've seen that type of interaction before but it was specific to Server 2003.

          • 2. Re: High memory usage on servers since upgrading to VSE 8.8 P6
            alhaawi

            hello wwarren

            is this related to Hyper-V only or it general, because we are planning to upgrade to patch 6, does it have issue with memory leaks?

            • 3. Re: High memory usage on servers since upgrading to VSE 8.8 P6
              wwarren

              Hi alhaawi,

               

              It looks like steveholleyuk is experiencing an issue that needs investigating.

              But like I mentioned in my reply, I haven't seen anything like it to-date with Patch 6 on hyper-v.

              • 4. Re: High memory usage on servers since upgrading to VSE 8.8 P6
                steveholleyuk

                Hi guys

                 

                Thanks for taking the time to reply. We have an open SR # <4-11256096511>, to which we have uploaded MER and log data. So far though no info from Support, I have tried calling but could not get through. Any help much appreciated, we have to manually restart these hosts overnight, and being as we have 200 servers in our Warwickshire schools - its a big problem. We could just uninstall the AV, but I would rather find a fix/reason to our issues? We had been running Agent 5.02/VSE P5 without issue ....

                • 5. Re: High memory usage on servers since upgrading to VSE 8.8 P6
                  wwarren

                  I took a look at your SR and the perfmon log provided; it showed the memory usage issue but was insufficient for showing the leak.

                  So there are 2 useful approaches to troubleshooting -

                  1. A perfmon capture over a longer period of time, suggested 60mins
                    1. Include Memory object,
                    2. Include Process object (this was missing from the provided .blg) - all instances please.
                      1. You can take samplings at intervals greater than 1s to reduce the size of the log, but try not to go longer than 15s.
                    3. The longer period of time will make it easier to demonstrate a leak occurring; the process object will let us see if its due to another entity leaking handles (which is commonly a cause for the symptom)
                  2. A complete memory dump, forced to occur when the issue is present
                    1. This requires configuring the system to save a complete memory dump, then reboot
                    2. Depending on how you trigger the dump a reboot may also be required for setting that up too.

                   

                  I made these recommendations in the SR that the technician can pass on to you also, since this is not a place to be used for Support tickets.

                  • 6. Re: High memory usage on servers since upgrading to VSE 8.8 P6
                    steveholleyuk

                    Hi wwarren

                     

                    Many thanks for your help, much appreciated. Sorry to miss-use the Communities for SR's, I posted because I was kinda hoping that we were no alone with this issue!

                    I will update the SR with the suggested data.

                     

                    Thanks

                    Steve H

                    • 7. Re: High memory usage on servers since upgrading to VSE 8.8 P6
                      alhaawi

                      Dear wwarren

                       

                      I would like to make sure that if there is no memory issue with vse p6, therefore I will start the windows task manager and watch the memory percent with time if it increases or not. Is this enough test to do?

                       

                      epo3pm.PNG

                      • 8. Re: High memory usage on servers since upgrading to VSE 8.8 P6
                        wwarren

                        Hi alhaawi,

                         

                        "Not really" is the quick answer.

                        Memory can be used directly and indirectly by applications; directly=reserving/committing memory, indirectly=creating objects (because those objects take up memory).

                        The operating system has different regions of memory that get used by applications and the OS. And it requires different tools to provide you visibility into these various memory segments.

                         

                        Task Manager is helpful if you want to look at the working set for a process. 32-bit and 64-bit systems have different max boundaries for memory usage of running processes, and you have to be careful when using Task Manager to determine a leak is present - is it a leak, or is the app just using a lot of memory?  Task Manager doesn't answer this question for you. But, if you've been watching memory climb without end and certainly if the max boundary usage is being hit, then there's probably a leak in how that app is managing its memory and/or resources.

                        There are better tools. And the best tool to use depends on what type of memory is being leaked that you are trying to troubleshoot. Poolmon and Perfmon are best for demonstrating a leak is occurring, because they provide data over time, and that also tells you the rate of the leak. Perfmon is especially useful when you include the Process object's counters, so you can see if other resources are being leaked (like Handle or Thread). And depending on the nature of the leak, a process dump or complete system dump may be necessary for getting to the root of the problem.

                        • 9. Re: High memory usage on servers since upgrading to VSE 8.8 P6
                          alhaawi

                          Hi wwarren

                          so up to your knowledge is there memory issue with vse p6? I noticed when I run epo task (Disaster Recovery Snapshot ) the memory jumps from 12% to 30%, when the task is completed the memory percentage stays at 30% and does not go back to 12!

                          1 2 3 Previous Next