9 Replies Latest reply on Jul 27, 2015 4:08 AM by M Bagheryan M

    Thousands instances of scan32/64 processes

    M Bagheryan M

      Thousands instances of scan32/64.exe processes appeared in our customer environment after  the VSE patch 4 automatically upgraded to Patch 5 (hosts are win7 ).

       

      The scenario is:

       

      The issue:

      1. all agents did updates of patch 5 on VSE systems (more than 600 pc).

      2. systems hung after  the multiplying process flooded of the available memory.

       

      our actions:

      3. we started to go in safe mode-->rename the scan32/64.exe-->restart the system

      4. we removed the patch 5 from epo master repository and also from system by epo then deploy the patch 4 on whole workstations

      5. after installation done we faced a new problem as Thousands instances of scan32/64.exe which is connected to ODS (on demand scan) ?!!!!!!

      6. there is no task or schedule for on demand scan in epo but it is happening.

      7. we changed the deploy way by adding << RUNONDEMANDSCAN="" >>in command line to not running on demand scan after finishing the installation. This wont work time to time.

       

      notes:

      1. Not all machines form the hostpark are affected, affection caused on about 80% of the systems.

      2. We also observe the same on patch4 machines.

      3. Comparison of not affected and affected machines bring the result :

      The machines that are working fine does not receive final part of microsoft updates (the list is attached).

       

      Still we have no conclusion or any idea what is happening.

      Did anyone encountered such issue ?

      Any ideas ?

       

      P.s.: It's mainly impossible to collect any data form the hung pc during the issue is starting. Time after start to total hanging takes seconds.

        • 1. Re: Thousands instances of scan32/64 processes
          tomz2

          I would strongly suggest opening a support case and gathering MERs from a sampling of systems.


          Support would be best suited to tell if it is for sure related to Windows Updates or not.

          • 2. Re: Thousands instances of scan32/64 processes
            M Bagheryan M

            We did already everything but there is no feedback yet.

             

            The thing I am looking for is:

            1. Are there any similar issue happened in other places?

            2. Are there any clue found about the reason of this issue?

            3. Are there any idea about this issue from more experienced engineers?

             

            We were just very lucky to collect webmer, one procmon log and GMER from the hung systems because in may be 20sec, systems were producing thousands of scan32/64.exe till death.

             

            There was a similar issue resolved by Patches provided by McAfee(McAfee KnowledgeBase - The VSE 8.8 On-Demand Scan uses large amounts of memory) but this issue happened after all this patches installed.

             

            Looking forward o here some genius idea about this case.

             

            Damaged systems are about 800 workstations.

             

            M.B M

            • 3. Re: Thousands instances of scan32/64 processes
              wwarren

              Couple ideas come to mind -

               

              It sounds like Early Launch Anti-Malware functionality (ELAM), which our product supports in Patch 3 and later and which Microsoft introduced with Win8 (but, some of the concepts of ELAM we can also support in older operating systems like Win7).

              When that code gets invoked it runs an ODS against any suspect driver. There will be an instance of Scan32/64 for each driver we're trying to scan.  Running Procmon with boot logging enabled with capture for you the details of the ODS, what parameters it's being started with.

               

              If you say the Windows update is the difference, perhaps the Windows update caused us to invalidate any existing data of what drivers are whitelisted causing us to re-examine. It also sounds like something is causing the ODS to fail to run properly... perhaps related to the update again.

               

               

              Test -

              • Disabling Access Protection. If that works get more granular; find which rule, when enabled, leads to the symptom then disable just that rule as a workaround. I would guess the "Prevent hooking of McAfee processes" rule. If that's what causes the ODS to get stuck then you've got a 3rd party intrusive app that the ODS doesn't like. Solveable if the 3rd party code is digitally signed; it can be trusted.
              • Disabling ELAM checks.  [HKLM\Software\McAfee\SystemCore\VSCORE], REG_DWORD “DisableELAMValidation”
                Set it to 1.  If that avoids the issue that's nice to know. The problem still needs to be solved though and we might need a device/VM to do that.

               

              If neither of these works, that's also good to know but will put more stress on needing an in-house reproduction to be able to investigate more effectively.

              • 4. Re: Thousands instances of scan32/64 processes
                M Bagheryan M

                HI William,

                Thanks for your brilliant advice.

                 

                I Will try this asap and let you know the result.

                 

                BTW, your note is really make sense in our case..

                 

                Appreciating.

                M.B M

                • 5. Re: Thousands instances of scan32/64 processes
                  M Bagheryan M

                  Hi William,

                   

                  Thank you vary much again.

                  Your suggestion does the trick. This is the way how we can stop the issue form repeating.

                  After creation of the mentioned registry key, the issue despairing.

                   

                  We will provide this info to support for future investigation. Hope they will figure it out.

                   

                  P.s.: It means we need to find some "unsigned" driver (for example) which persist on all systems and causing windows to call on-demand scan of VSE?

                  The cause should be very specific for that particular infrastructure...

                   

                  Cheers.

                  M.B M

                  • 6. Re: Thousands instances of scan32/64 processes
                    wwarren

                    P.s.: It means we need to find some "unsigned" driver (for example) which persist on all systems and causing windows to call on-demand scan of VSE?

                    The cause should be very specific for that particular infrastructure...

                    That is a possibility.

                    My fear is the behavior could be due to a broader systemic problem within the VSE product. There isn't enough data to say which scenario it might be.

                    I mentioned using Procmon with bootlogging, because the hope is if we can identify the command line parameters being passed to the Scan32/64 processes we'll have a better idea on if it's a case of some "unsigned" drivers, or if perhaps our own cached validation data is invalid, or something else.  It would be very challenging to investigate this type of symptom without having a VM or local reproduction.

                     

                    Knowing that workaround option is successful is good to know though. Thanks for sharing.

                    • 7. Re: Re: Thousands instances of scan32/64 processes
                      M Bagheryan M

                      wwarren wrote:

                       

                      That is a possibility.

                      My fear is the behavior could be due to a broader systemic problem within the VSE product. There isn't enough data to say which scenario it might be.

                      I mentioned using Procmon with bootlogging, because the hope is if we can identify the command line parameters being passed to the Scan32/64 processes we'll have a better idea on if it's a case of some "unsigned" drivers, or if perhaps our own cached validation data is invalid, or something else.  It would be very challenging to investigate this type of symptom without having a VM or local reproduction.

                       

                      Knowing that workaround option is successful is good to know though. Thanks for sharing.

                      Hello again Dear William,

                      Following your suggestion about Procmon with boot logging,

                       

                      I just did it on x86 system today and uploaded it on the portal which I wrote you on Private message.

                       

                      I would appreciate your kindly attention before.

                       

                      Cheers,

                      M.B M

                      • 8. Re: Re: Thousands instances of scan32/64 processes
                        wwarren

                        Following your suggestion about Procmon with boot logging,

                         

                        I just did it on x86 system today and uploaded it on the portal which I wrote you on Private message.

                        I saw it. Thanks.

                        There was evidence that system was in a bad state. DLL versions being loaded by Scan32 were not the expected version. Something had happened to the Patch 5 install attempt (actually "attempts" as there were multiple) which left a mix-match of DLLs.

                        In that unexpected state all bets are off as to what kinds of issues you're going to run into. This ODS spawning multiple instances would've been just one of potentially many problems.

                         

                        I suggest steering focus away from "Why is ODS running multiple times on startup" back toward the root of the problem which would be "How did we end up with a mish-mash of DLL versions?" and then of course, "What can Intel Security do to avoid that?".

                        But, if the goal is simply to get the systems back to a healthy state, I imagine it'll require some removal of product(s) to then install a clean base. And assuming there's nothing present to confuse or otherwise mess up the installation, you'll be OK.

                        • 9. Re: Thousands instances of scan32/64 processes
                          M Bagheryan M

                          Hi Williams,

                           

                          The SR is updated.

                          Unfortunately there is no possibility to send you requested VM.
                          Reasons:
                          1. Customer have no licensed software to convert host os to VM.
                          2. Customer just refused it , in terms of their privacy policy. Because systems may contain (somewhere inside) the classified information. This is the issue which is out of our control.

                           

                          Cheers.

                          M.B M