1 2 Previous Next 17 Replies Latest reply: Nov 26, 2012 7:06 PM by Peacekeeper RSS

    McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)

    beheer

      1) Memory leak (The Nonpaged-pool fills up):

       

      More than two weeks ago, after the update of all our (64-bit) Windows 7 machines, to version 11.6(SecurityCenter) and 15.6(VirusScan), excessive memory leaks occur and the scans are very slow.

      After 5 to 8 days uptime (depending on the IO use), the machines crash with a BSOD(Blue Screen of Death), because the non-paged memory reaches the physical memory limit.

       

      On the XP machines, 9 in total, (Version 11.0/15.0/..) these problems are not seen at all! (The *.6 seems to be a Win7 only version.)

       

      The newer versions v11.6.434(SC) and v15.6.231(VS) seemed to improve the memory issue a bit, the machines build up their Nonpaged-pool slower, but did not fix the leak.

      The leak occurs from the kernel-driver FWPKCLNT.SYS, in c:\Windows\Sysnative\drivers\. (On my systems it has date 2012-08-22 20:12 (UTC+1), is version: 6.1.7601.17939, and is installed on 12-September-2012, the time from the MS-Updates batch.)

      This was found out using 'poolmon.exe' from the Windows Driver Kit. (The driver with Tag 'Fwpx', never Frees any! of its allocated memory and when the nonpaged-memory reaches a total of about 3GB it crashes our 4GB PCs. (Max nonpaged memory is 75% from the Physical Memory size on a Win7 64-bit machine.)

      See attachement A.)

      I have about 5 almost 100% identical BSODs. (All report a crash in the driver NETIO.SYS (BugCheckCode=1d, String=DRIVER_IRQL_NOT_LESS_OR_EQUAL).

      See attachment B).

       

      If the machines are rebooted, before the Non-paged memory reaches its border (the Nonpaged Limit), the machine does not crash.

       

      Filling up almost all the Physical memory with nonpage-able pages, of cause really hurts the performance of a PC.

      This is surely (one of the) reasons the next problem also occurs.

       

       

      2) Slow Scanning (either Scheduled, Full or Manual):

       

      From the Windows 7 PCs, all with the latest patches (so including the patches of yesterday 2012-Oct-11), 6 machines were tested.

      The worst behaviour is shown on 3 (of those 6) PCs. They have a dual core processor (and only 4 GB of memory).

      Also their scanning, which normally lasted maximally up to 4 hours (before this was even 2 hours!), now sometimes takes over a day or longer!

      (This is even with the "Scan using minimal resources" setting turned off.)

       

      The Resource Monitor (Disk-tab) shows the scan has problems with big (ASCII) files. The INBOX and Sent files from Thunderbird take a very long time to scan. During this the non-page memory size raises even more quickly.

      The files are in plain ASCII text (not compressed or anything), and its size reaches up to 3.5GB for some INBOX-es. As said before, they are scanned a lot faster on the slow XP machines!

      The slow scanning is occurring since it was first notices on 26-September. (We run weekly scans).

       

       

      Notes/Conclusions:

       

      The described problem, the memory leak, is also very extensively reported in at least 2 other threads in the forum.

      It seems to be the EXACT same problem for which McAfee (until now) has not found a solution.

       

      - One link which mentiones the exact same problem is the one from user Fragged, posted on 19-September:

      (See: https://community.mcafee.com/thread/48661?tstart=0)

       

      - It is not related only to machines with lots of IO, as the Blue Iris Camera-PCs from user Lotty.

      (See: https://community.mcafee.com/thread/47463?tstart=0)

      He already reported this problem as early as 7-Augustus!, yet there is no fix!

       

      - It is not limited to Total Protection, as we are using VirusScan Plus only!

      It seems to be in the core-software of the newer *.6 version for Windows 7 PCs.

       

      - There must be a lot of unaware users from McAfee out there, who probably reboot their machines regularly, and therefore may not even notice the problem.

      (Or it is simply too much hassle to write here...?)

       

       

      McAfee should give this issue some very, very serious attention, because they are loosing customers this way!

       

       

       

      Attachments:

       

      A) Nonpaged Memory-pool usage shown by Task Manager, RamMap(sysinternals) and poolmon.

        Poolmon also shows that 0 Frees were done in the 4-days the 'Fwpx' driver build up its nonpaged pool to 2.5GB ! (NP-total=~2.9GB)

       

      B) BSOD from this morning 2012-Oct-11, shown with BlueScreenView (It also shows one from 2 weeks ago!)

       

      Message was edited by: beheer on 10/11/12 8:11:49 AM CDT
        • 1. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
          lotty

          Based on my direct conversations with the McAfee engineering team  …  they know they have a problem with a memory leak ... it tends to present with any significant IO event .. further the rumor is a fix has been completed but not yet released … antivirus programs .. my sense is Microsoft changed a few things in some OS updates released back in July .. McAfee did not see this coming or just did not test thoroughly enough .. McAfee released their own new build right after this July batch of Microsoft updates but failed to incorporate the changes that other antivirus vendors were able to navigate.

           

          Why hasn’t McAfee released the fix .. only they know .. but I do think they owe it to the user community to COMMUNICATE  .. .. about a month ago just prior to the release of the current builds  SC 434 / VS 231 I was told about the pending  fix .. based on my testing which I have posted in my link (see link below)  .. the current build was a  SLIGHT improvement  to the timeline of the crash event but did not change the outcome. So my conclusion was they thought they had a fix but it did not work.

           

          IF IF IF this is the case, McAfee needs to go back to the drawing board on how they test new releases. I sat  with their engineering team many times for hours while they remoted into my PCs and captured real-time logs of the traffic and crashes. McAfee documented many crash events .. throughout the process the same engineer would call me and offered insights into the current status. I have not heard from him in over a month. He did call me a couple of days before the release of the current builds to suggest I install and retest.

           

          My understanding of root cause relates to some function calls into the kernel specifically for

           

          Anyway that is my best understanding of the situation. Any yes one can only wait it out or switch. What a complete pain.

          Other links covering the same issue

           

          https://community.mcafee.com/thread/47463?tstart=0

          https://community.mcafee.com/thread/48661?tstart=0

          • 2. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
            Ex_Brit

            This is what I have received today from support and I quote:

             

            The bottom line is that this is a Microsoft problem caused by their fwpxclnt component not releasing memory consumed in Win 7, SP1 (Windows 8 doesn’t experience this problem). 

             

            This is a distilled version of our findings:

             

            ISSUE: Windows 7 SP1 runs out of memory because the nonpaged pool fills with buffers tagged Fwpx.

             

            From msdn:

            "Remarks

            The FwpsNetBufferListRemoveContext0 function asynchronously removes a tagged

            context that was previously associated by calling

            FwpsNetBufferListAssociateContext0. It is not required to use this function,

            because the tagged context will be removed automatically when the packets move

            through the stack. This function is provided so that a callout driver can stop

            processing in unusual situations."

             

            which clearly states that netio/wfp should auto-remove associated contexts.

            This is apparently not happening on win7.

             

            RESOLUTION: The network filter driver has been revised to prevent Windows 7 from allocating these buffers.

             

            (We should mention the Fwpx pool tag because it's a distinguishing feature a user might discover while troubleshooting with Poolmon or other tools.)

             

            Despite the fact this problem is *caused* by a Microsoft component, we are adjusting our software to compensate.  The change has been made and will be included in the next release of the Core components – I still haven’t heard back which specific version number that will be in, nor do I yet know when it will be released.

            • 3. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
              beheer

              Hi Ex_Brit,   (or actually, it should better be addressed to the McAfee management!)

               

              Thanks for your clear answer. At least McAfee is nog agreeing that it is a problem with their software.

              The bad thing is only that it took over TWO MONTHS. On 7-Augustus Lotty already reported the problem!

               

              It is hard to understand that the McAfee people of the Forum do not have better backing from the technical people in the back-office.

              How hard is it to give an estimate when the fix will be released?

               

              Or why can we not 'downgrade' to the 11.0/15.0 release from McAfee???

              Why is there no work-around yet?

               

              Reading Lotties and Fraggeds threats, the McAfee engineers agree that a lot of customers are having this problem.

               

              I realy start to wonder if McAfee takes its customers serious.

               

              I did (again) spend many hours on the phone with McAfee technicians (like Lotty and Fragged).

              I read the forum, and investigated myself, which in the end is propably cheaper.

               

              If the front- and backend-office of McAfee worked better together, a lot of useless hours could us customers be spared!

               

              Just make a thread in the forum with: "Known problems" for each separate Product.

              This information should come from the back-office though! McAfee to tell their customers that they are aware there is a problem with one of their products;  that they ARE working on it; and give us update what the status/progress is on that specific problem.

               

              To keep us in the dark with: "I do not know when it will be released", simply does not cut it!

               

              Again, we need!

              - a quick fix, or

              - roll back possibilities to be able to go to the older (good working) version!   (Before it worked!!!)

              - otherwise a work-around.

              - information about a known issue with their software

              - information on when it will be fixed, or at least how things are progressing.

               

              Please start to take us customers serious!

               

              Message was edited by: beheer on 10/11/12 3:53:46 PM CDT
              • 4. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
                lotty

                FYI bobcat posting yesterday on my posting

                Got another new version tonight (11.6.435 / 15.6.231) and the problem is NOT fixed.

                 

                postings that seem to be same issue are

                https://community.mcafee.com/thread/47463?start=90&tstart=0

                https://community.mcafee.com/thread/48661?tstart=0

                https://community.mcafee.com/thread/49170?tstart=0

                • 5. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
                  beheer

                  Hi Lotty!

                   

                  You are right it is not fixed.

                  Nobody seems to know what this 11.6.435 update is for, as McAfee indeed does not have release-notes (At least not for us customers).

                   

                  I also wrote something in your thread. And tried to get an answer through the Mods from McAfee, as I understand now they are closer to the fire, but are not McAfee employees...

                   

                  I hope they will start to take us/our problem serious...

                   

                  Message was edited by: beheer on 10/19/12 6:31:47 PM CDT
                  • 6. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
                    outtasight

                    Figures.

                     

                    Sometime in early October I started getting problems with a Win7 32 bit machine with 2GB RAM.  This machine is on 24x7 as a data logger and CCTV recorder.  Until then, this machine was rock solid with up-time measured in weeks (reboots only occasionally done for MS updates being applied).

                     

                    It started doing weird things with the machine slowing to a crawl and eventually crashing with a BSOD.  I stopped using the CCTV software for a while and it just took longer to crash (a few days).  If I caught it in the death throws, I could see the free memory stat counting down like a bomb with the disk monitor showing the system process thrashing the disk with reads and writes.  If rebooted before zero memory free, it would recover fine.

                     

                    I'm on the VirusScan Plus 15.6.231 and Security Center 11.6.435 with up to date MS paches.

                     

                    The usual pattern of: MS patches core system component... AV vendor doesn't get notified or doesn't pay attention... PC gets smegged... MS and AV vendor point fingers at each other and bury heads in sand...

                     

                    Definitely McAfee software needs to be able to roll back to a "Last known good" config...  Unless that version has an exploit that would allow malicious virus code to cause crashes, data loss and pain worse than the crashes, data loss and pain that the patched version of the AV product is causing me now.

                     

                    At least AV vendors malicious code comes with an uninstall routine

                     

                    Message was edited by: outtasight - got the version numbers the wrong way round on 13/11/12 01:40:44 CST
                    • 7. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
                      Peacekeeper

                      Actually the fix is in 12.1 released late last week . If you uninstall mcafee and reinstall it you will get the fix. For updates it is I assume throttled and the update will take a while to come.

                       

                      Though 1 users says (build SC 11.6.435 / VS 15.6.231/ PFW 12.6.186) fixes it.

                       

                      read the last 2 pages of

                      https://community.mcafee.com/message/263254#263254

                       

                      If you are going to uninstall do it this way

                      http://service.mcafee.com/FAQDocument.aspx?lc=1033&id=TS101331

                      reboot and run

                      http://download.mcafee.com/products/licensed/cust_support_patches/Mcpreinstall.e xe

                      reinstall from your account

                      • 8. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
                        outtasight

                        Thanks.  I'll see what 12.1 brings.  I don't have time in the next couple of days to play with the machine so maybe it will get the update automatically.  Otherwise I'll try to re-install McAfee at the weekend.  I was going to dig out the last known good image backup of the machine, but if 12.1 fixes it, that will save me re-imaging the whole machine.

                         

                        SC 11.6.435 /  VS 15.6.231  definitely were associated with a couple of BSODs and near misses (yesterday was a near miss with the machine chugging heavily with only 4MB free RAM remaining).

                         

                        Selecting "update now" doesn't find anything other than todays definition update.  Another product I used had a separate "update product" option to force it to look for new software core updates as opposed to just definition updates.

                        • 9. Re: McAfee AntiVirus Plus 2012, Memory leak in FWPKCLNT.SYS (and terribly slow scans!)
                          Peacekeeper

                          Oh the update is only available from the US Uk Ca and Au servers so if you are reinstalling set 1 of those as the country.

                          1 2 Previous Next