cancel
Showing results for 
Search instead for 
Did you mean: 
rgijsen
Level 8
Report Inappropriate Content
Message 1 of 38

VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

Hi,

the subject says it all actually. I had to track down high CPU load in our environment. Practically all machines are Hyper-V guests. but we also have some physical machines running, some of which have VSE installed. All VSE installations are managed through epo. The machines are either 2008R2 or 2012R2. All show the same behaviour: When VSE 8.8p4 is installed, network traffic generates as much as100% cpu on one thread (I think always on CPU0). This slows down the machines tremendously. This is not so obvious with small transfers, but when we transfer an .ISO file or another large file, the machines become really slow. Traffic is as fast as 80/100MB/sec (gbit backbone). The CPU load is not 'bound' to a process when I look in taskmanager, but rather is it Kernel time. So I suspect some low-level driver being the culprit. Now when I uninstall VSE this issue is completely gone. CPU load with large traffic is about 10-20%, and I get the full 120MB/sec (gbit) without any issue.

One might say 'exclusions, network scanning and archive scanning'. All that has been covered, those are disabled and it didn't help. In fact, I disabled ALL McAfee services on a few testnodes, and CPU was still very high. As no services are running, no McAfee processes are running (atleast not visible to me). Uninstall VSE and it's running perfectly again. This is 100% reproducable here on both virtual machines as physical ones, and is a serious issue for us.

So in short, with VSE installed but even when completely disabled, very high CPU load with network traffic. VSE uninstalled: all systems running fine. Anyone has a clue, can we disable a low-level-network driver (I don't need network traffic to be scanned, I need files to be scanned when they are created or read)? I found no KB articles on this whatsoever.

To show what happens two screenshots, the first one with VSE installed but both source and destination location and network excluded from scan AND VSE services disabled (so no mcshield.exe running). Second screenshot with VSE uninstalled. First machine running at about 80MB/sec, second machine at 120MB/sec.

with vse.png

without vse.png

1 Solution

Accepted Solutions

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

McAfee is fully aware of the issue since May.
They have "alpha" (POC) driver and should prepare HF or include it in one of the next patches.

Please report your issue, you can mention my SR 4-5606236293.

37 Replies
bleeder
Level 9
Report Inappropriate Content
Message 2 of 38

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

If you only disabled the McAfee services in the list of Windows services, then McAfee isn't really disabled.  There are still filter drivers/etc that are active.  You can disable/re-enable them using the mfehidin command.  For reference see McAfee KB56059.

Here is another thread that may be related:

https://community.mcafee.com/thread/66199

ansarias
Level 13
Report Inappropriate Content
Message 3 of 38

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

Try to run process monitor tool to see what is happening on server to make high CPU usages. Based on results you can do exclusion to make CPU utilization at normal level.

OR

McAfee On Access scanner policy, remove scan inside ZIP files + Remove scan on files opened for backup + Remove scan on Network Drives. Using these options will make CPU normal.

rgijsen
Level 8
Report Inappropriate Content
Message 4 of 38

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

@ansarias Thanks but you did not read my post really well I guess. I did exclude everything. I even disabled all services. So except for some low-level drivers, nothing is running anymore. That means excluding from mcshield will not do the trick.
I'll still see what procmon will tell me but probably it's just 'system' that's using all cpu.

@bleeder Thank you as well, I'll check that out.

I wonder if noone else is having these issues. We are only having a 1Gbit backbone here, but imaging how bad this is when running on 10Gbps.Even when excluding everything,

bleeder
Level 9
Report Inappropriate Content
Message 5 of 38

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

Use Process Explorer, right click the process using the most CPU, click the Threads tab.  Then sort the threads by CPU.  That should help you pinpoint the issue.

Highlighted
pato
Level 7
Report Inappropriate Content
Message 6 of 38

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

This could also be because of a bug in the network card driver. Try to update the driver, it might be fixed afterwards.

[edit] removed the question regarding Patch as it is already stated that it's P4.

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

Try disabling the McAfee Mini-firewall device driver (MFEWFPK.SYS), as long as you are not currently making use of port filtering via VirusScan Access Protection. To disable that driver simply rename it (e.g. MFEWFPK-disabled.SYS) and reboot.

I have a case open for what appears to be the same issue. It does not happen on physical servers but only on Hyper-V and VMware VMs. I ran a before and after test to be sure, where I would copy a large file to the server and time how long it takes, as well as watching the CPU usage.

With the evidence I sent to McAfee in my case, they have agreed that there is a problem and they are now moving it to McAfee Engineering.

The way I look at it, the mini-firewall driver is shimmed into the network stack (as a mini-filter driver) -- in order to do its work -- and should not consume any cycles at all if nothing has been configured for it to filter/scan.

I am hopeful of a hotfix but in the meantime am considering disabling the driver. Imagine the performance boost!

rgijsen
Level 8
Report Inappropriate Content
Message 8 of 38

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

Hi,

I've just 5 minutes ago finished a testsetup to create a case with McAfee regarding this. Would you want to share your case ID so I can refer to that, to speed things up probably?

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

Would be curious to see your test scenario. My case # is 4-6523877545. I have not done much testing on P4. My tests were comparing P2 and P3. I am guessing that there was a change introduced in P3 that continued into P4, which is causing this.


There is an interesting change in mini-filter driver architecture by Microsoft in Server 2012 (Win8). Check out these articles:


http://www.osronline.com/showthread.cfm?link=242949

http://notesfromthedatacenter.wordpress.com/

http://blogs.technet.com/b/askcore/archive/2012/03/18/fixed-cluster-shared-volumes-csv-in-redirected...

http://msdn.microsoft.com/en-us/library/windows/hardware/dn265170%28v=vs.85%29.aspx

Offloaded Data Transfers (Windows Drivers)

The last one adds an interesting wrinkle for Hyper-V (Offloaded Data Transfers).

Re: VSE 8.8 p4 - High CPU load on network traffic even if all McAfee services disabled

Jump to solution

My scenario is pretty simple - I've setup 2 2012R2 VM's, both updated by WSUS to current patchlevel, one with VSE8.8 p4, one without it. The difference especially with large files is astounding. Disabling mfewfpk.sys indeed stops the issue, so for us it's reproducable and narrowed down to that filterdriver, while we don't actually use it. I registered a case, still waiting for initial response. Same happens in 2008R2 VM's by the way, which don't support ODT as far as I'm concerned. I don't run VSE on my Hyper-V hosts, as excluding all Hyper-V storage as a best practice (and boy did I have issues with VSE running on these hosts even with those exclusions) means it's not doing anything really functional anyway. With regards to network drivers; there is not really such a thing as they are updated by installing the guest-additions and they are all up to date.

More McAfee Tools to Help You
  • Subscription Service Notification (SNS)
  • How-to: Endpoint Removal Tool
  • Support: Endpoint Security
  • eSupport: Policy Orchestrator