4 Replies Latest reply on Apr 22, 2012 2:29 PM by hatnell

    VirusScan Enterprise leaks nonpaged pool

      VirusScan Enterprise 8.8 32bit (with scan engine 5400.1158) currently uses ~1GB memory from the nonpaged pool. That’s half of the physical memory of the system.

       

      McAfee VirusScan Enterprise.png

      (RAMMap and Poolmon)

       

      After 12 hours, during which there were little activity, the memory picture remained almost unchanged.

       

      OS: Windows Server 2008 SP2 32bit

      System uptime: 9 days

       

      for %i in (mfe*) do @echo %i & sigcheck -q -n %i

      mfeapfk.sys

      SYSCORE.14.3.0.514

      mfeavfk.sys

      SYSCORE.14.3.0.514

      mfebopk.sys

      SYSCORE.14.3.0.514

      mfeclnk.sys

      SYSCORE.14.3.0.514

      mfehidk.sys

      SYSCORE.14.3.0.514

      mferkdet.sys

      SYSCORE.14.3.0.514

      mfetdi2k.sys

      SYSCORE.14.3.0.514

       

      Note that such memory leakage cannot occur on 32bit WindowsXP/2003 because the nonpaged pool of these OSes is restricted to 256 MB or less.

       

      Is this a know VirusScan issue?

      The system is still alive. Is there a way to see the memory content of MFE0?

       

      References:

      http://blogs.technet.com/b/markrussinovich/archive/2009/03/26/3211216.aspx

        • 1. Re: VirusScan Enterprise leaks nonpaged pool
          sbenedix

          Hmm, an update should be the first thing to consider, you are running on an old set of  drivers (8.8 RTW), P1 for VSE 8.8 will update you to 14.4.0.457. Note issue 15 in the P1 readme. From the data presented, difficult to say what is going on and what is at the root, that would require perfmon, poolmon, dump analysis. I'm quite confident though that P1 will alleviate the symptoms you describe. Make sure the OS is up to date (Windows update).

           

          The numbers are odd though.

           

          More reading if you are interested, I found useful: http://blogs.msdn.com/b/ntdebugging/archive/2006/12/18/understanding-pool-consum ption-and-event-id_3a00_--2020-or-2019.aspx

           

          You would be able to analyse MFE0 memory consumption with a memory snapshot (kernel dump) while the machine is in the state of exhibiting the symptoms.

          • 2. Re: VirusScan Enterprise leaks nonpaged pool

            I found a way to dump all pool allocations tagged with MFE0 without restarting the system.

             

            1. Create a memory dump (memory.dmp) without restarting the system - https://kc.mcafee.com/corporate/index?page=content&id=KB50467
            2. Download PoolTools from http://computer.forensikblog.de/en/2007/11/pooltools-version-130.html
            3. Open poolfinder.pl and change the line 'usage|?|' => \$opt_usage, to 'usage' => \$opt_usage,
            4. Repeat the previous step for PoolDump.pl
            5. perl poolfinder.pl --dbcreate –dbfile Pool.db3 --nolisting memory.dmp
            6. perl PoolDump.pl --dbfile Pool.db3 --dumpfile memory.dmp --tag MFE0 > MFE0.txt
            7. Open MFE0.txt in text editor which does not try to load the whole file into memory (I used TextPad)

             

            Notes:

            Step 5 and 6 require perl with necessary packages installed (I used ActivePerl). These two steps may take several hours.

             

            These instructions probably are far from optimal, but they give freedom to everyone technically literate to follow them, to obtain an approximate idea of what is going on without relying on the interpretations of someone else.

             

            In MFE0.txt, which I generated by following essentially the same instructions, I found a lot of "cygwin" strings. Recently I compiled OpenSSL in Cygwin environment in order to create self-signed certificates.

             

            On the basis of that information I created a test case which clearly illustrates the memory issue:

             

            1. Install Cygwin
            2. Download openssl-1.0.1.tar.gz from http://www.openssl.org/source/
            3. Extract the archive and install all Cygwin packages, which are necessary for compilation of OpenSSL
            4. Start Poolmon, sort the list by Bytes (b) and save a screenshot:

            McAfee VirusScan Enterprise - before.png

            1. Open Cygwin console and enter into the directory of OpenSSL
            2. There execute: ./config && make && make test
            3. If the compilation and tests are successful then close Cygwin. Otherwise fix the compilation problems, extract OpenSSL into another folder and return to step 4.
            4. Wait 20 minutes and save a screenshot of Poolmon:

            McAfee VirusScan Enterprise - after.png

             

            Although the test case involves compilation of source code the underlying memory problem may manifests itself in other situations.

             

            > I'm quite confident though that P1 will alleviate the symptoms you describe.

             

            I don't see a clear reason to think so. Issue 15 in the release notes of Patch 1 refers to the process validation service, which aparently is part of Access Protection. Access Protection is not installed on the system.

            • 3. Re: VirusScan Enterprise leaks nonpaged pool
              wwarren

              Hi Hatnell,

               

              You should check to see if you have a process that's leaking Handles.
              We have seen and addressed NPP leak issues in our drivers in previous releases, but we haven't seen anything for a long time that has been confirmed as a McAfee issue - instead what we've found is we are a victim of a poorly behaving process, who is opening file handles endlessly (which we are having to track in our driver code, thus using up memory).

               

              Otherwise, you should force the system to dump (or you can try live dumps if your system must remain up, but live dumps aren't 100% reliable to have the data we'll need - or at least the data may not be coherent). With a dump we'll be able to track the pool usage and probe for other potential causes.

              • 4. Re: VirusScan Enterprise leaks nonpaged pool

                ESET NOD32 Business Edition 4.2.76.0 passed the test case without leaking memory.

                 

                After uninstall of McAfee VirusScan Enterprise 8.8 (and McAfee Agent) and subsequent system restart, part of McAfee remains active and occupies RAM memory:

                McAfee VirusScan Enterprise after uninstall.png

                 

                One way to remove that part is the following:

                1. Start -> Run -> cmd
                2. sc delete mfetdi2k
                3. del %SystemRoot%\system32\drivers\mfetdi2k.sys
                4. Restart the system