1 2 Previous Next 11 Replies Latest reply on Apr 29, 2010 3:30 PM by andymease

    Mcshield & VsTskMgr pegging CPU 100% on Citrix

    passmaster16

      Hi,

       

      I am running Citrix Presentation Server 4 on Windows 2003 Server SP2 (32 bit) with VirusScan 8.7 Patch 1.  We are also using EPO 4 to manage these servers.  We have been seeing over the past week or so instances where either Mcshield or VsTskMgr will spike the CPU on the server to 100%.  We noticed that it has been doing this when a dat file update is being applied.  When this occurs, the CPU spikes very high to the point of making the server unresponsive.  Sometimes it will last for a few minutes up to 20 minutes after the update is performed.  Over the past few days it's been fairly consistent as I can see performance issues occur around noon when the dat update is applied.  So far McAfee support has advised to either upgrade to patch 2 and if that doesn't work, reschedule the dat updates to occur during off peak hours.  Has anybody else seen this and do you have any suggestions on resolving the issue.  Like I said, it seems to have started with the last week or so of dat updates.

       

      Thanks in advance!

        • 1. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
          maziz

          Hi

           

          I would recommend that you run Process Monitor from Microsoft, which can be downloaded from HERE. Run this whilst you are seeing the spike in CPU from McShield.exe and VsTskMgr.exe

           

          This should give you a much better insight on what these two processes are doing whilst using the CPU. You can use Process Monitor to filter the results to only show u McShield and VsTskMgr.

           

          Do you see VsTskMgr making multiple requests to .BUP files in C:\Quarantine ?

           

          Can I also suggest that you take a look at the VirusScan Quarantine folder. Normally by default this folder is C:\Quarantine. How many items are there in this folder?

           

          If there are many items, is it necessary for them to be kept or can they be deleted? Or is it possible to make a backup of this folder and then delete its contents?

           

          The reason I am suggesting this is because every time a DAT update takes place, the Quarantine folder is queried by the new DAT for false positive detections. This could be the possible cause if there are too many items in this folder.

           

          Hope this helps.

          • 2. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
            passmaster16

            Hi,

             

            I am still seeing issues on my Citrix servers with Mcshield consuming high CPU resources.  It appears that the problem occurs during post update when the on access scanner is enabled.  I am seeing 5-10 minute slowdown while Mcshield uses these CPU resources.  I was able to reproduce the behavior by manually stopping/starting the OAS per this thread http://community.mcafee.com/thread/4756  I disabed the "process on enable" feature of the OAS in EPO without much improvement in performance.  I'm seeing the issue on both VSE 8.7 Patch 1 and Patch 2.  My guess is that there is some link to the fact that the server is Citrix/Terminal Services box with multiple users.  Perhaps this is interacting with the OAS and making the problem worse.  The Mcshield service is using about 30 MB of memory at peak but the CPU is also high when the problem occurs.  What else would Mcshield/OAS be scanning when the service is restarted?

             

            Thanks

            • 3. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
              wwarren

              We have recently identified an issue here (just today, actually!).

              Where the email scan module is being used, in VSE 8.7i, it is possible to get into a state where our EngineServer service is maintaining multiple instances of the scan engine (and consequently, multiple copies of the DAT files). The immediate symptom is bloated memory usage of the EngineServer process. We have extrapolated from this symptom too that it can have impact on McShield and VsTskmgr.

               

              The workaround would be to restart the EngineServer service. This will drop the memory consumption to normal levels, and improve the time taken during updates.

              The time lag comes into play because of the multiple engine instances, each having to be initialized. It will account for many minutes during the post-update phase.

               

              Once a hotfix has been confirmed for this issue I will be pushing to have it made available via our Knowledgebase, so everybody can get it without need to contact McAfee Support.

              Additionally, 8.7.1 will have further improvements to lessen the system impact during updates.

               

              Kudos should be given to those customers who were able to work with McAfee Support to gather quantifyable data that lead to discovering the cause. This issue was hidden by others issues that had been solved in earlier patch releases. Now it has been brought to light, and a solution is on the way.

              • 4. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
                passmaster16

                Hi William,

                 

                Thanks for your reply.  One thing that I've noticed is that when a server runs for a long time without rebooting, it appears that the Mcshield process doesn't always want to release the memory when the dat file is updated and OAS restarted.  For instance, the peak memory usage for the Mcshield process on our Citrix servers is usally around 30 MB.  On the servers that have rebooted recently, the memory will be released during the dat file update process which drops the memory usage down to about 8-10 MB.  Then over time this usage goes back up as more users connect to Citrix.  This behavior does not occur on servers which have been up for a few days or weeks at a time.  The memory on these servers stays at 30 MB even though the dat was updated and OAS was restarted.  I'm not sure if this plays into the issue.  Also, while I'm not surprised by this, the slowdown caused by the Mcshield process is much worse on our single core server systems than on our quad core systems.  When the dat update runs on these systems, they are pretty much unsable until the update and post processing is completed.  The quad core systems can process the update in the background much better and still allow our users to work in the Citrix enviornment.

                 

                The bigger problem we have right now and probably more troubling is that we've been experiencing random lockups on these servers a few hours after the new dat files are applied.  When these lockups occur, the systems are not accessible via Citrix or RDP.  They lockup so hard that they quit writing events to the event viewer.  The only way to get them working again is to power down and restart.  When we examine the event logs for errors, we cannot find anything other than the last event to be written is usually the McAfee dat Update notification.  We believe that VirusScan is playing a role as we never experience these lockups during the day hours -- prior to the dat update process. We've had the lockups with both patch 1 and patch 2.  We are running EPO 4 with agent version 4 build 1444.  Some people on these forums have mentioned that Global Updating feature of EPO has caused lockup issues.  Since we did not want to go to the trouble of disabling Global Update and scheduling the updates through tasks, we decided to completely remove the EPO agent from our Citrix servers in hopes of isolating the problem to EPO.  We did this last week and so far we've not seen the lockup issues.  We have VSE scheduled to simply update from the Internet for the time being.  Time will tell if we can find a correlation between Global Updating/EPO and VSE causing Citrix lockups.  We don't know why there would be any difference of updating via the VSE built-in task vs using EPO Global Updating but since we noted the problems mentioned in other threads, we decided to investigate the issue from this angle.

                 

                Since we have not encountered either of these issues on our Windows XP workstations or our other Windows 2003 (non-Terminal Servies/Citrix), we feel that whatever issues we have are related to the Citrix multiuser environment.  We even excluded directories per the Citrix technote http://support.citrix.com/article/ctx114522 without any success.

                 

                Thanks

                Josh

                • 5. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
                  D-Fens

                  "

                  Once a hotfix has been confirmed for this issue I will be pushing to  have it made available via our Knowledgebase, so everybody can get it  without need to contact McAfee Support.

                  Additionally, 8.7.1 will  have further improvements to lessen the system impact during updates.

                  "

                   

                  when will this update approximately be available? i would highly appreciate any improvement to the update process!


                  maybe with the release of agent 4.5 p1 (mid-february)?

                  • 6. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix

                    I am also facing the same issue. I am using a Dell Latitude D630 Laptop, Win 7 McAfee 8.7i. At times, EngineServer.exe, VsTskMgr.exe or McShield.exe starts consuming 100% of CPU which makes system completely non-responsive. This lasts for several minutes.

                    • 7. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
                      wwarren

                      We are still evaluating the proposed code change in the field. It's too early to project a date since we're still discovering if we've got the right solution.

                      Initial responses are positive, however, but it's still uncertain whether the single issue we identified will have a cascading effect to the multiple symptoms people seem to experience. That, would be nice.

                       

                      For the Citrix hang mentioned, that sounds very much like an interoperability issue between McAfee and Citrix drivers.

                      You should configure the system to save a kernel or complete dump, and implement the registry tweak to crash it with the CTRL+SCROLL(x2) keyboard sequence. Crash the system when it is not responding and submit the dump file to both McAfee and Citrix (it will help your cause if you are running the latest of both vendor's software).

                      To date, interop issues involving Citrix have needed code changes from the Citrix folk.

                      • 8. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix

                        Any update on this issue? Is there a patch? I was forced by hq to update my lappy last week (8.7.0i engine 5400.1158) and have already experienced same issue several times. One soln posted above is to restart engine. What is best way to do so? Many thanks.

                        • 9. Re: Mcshield & VsTskMgr pegging CPU 100% on Citrix
                          passmaster16

                          I haven't seen the issue on Citrix since removing the ePO agent from those servers so that has been my workaround.  I'm just running VSE 8.7 Patch 1 in standalone mode for now.

                          1 2 Previous Next