1 2 Previous Next 13 Replies Latest reply on Oct 23, 2014 10:40 AM by justav

    Serve performance degradation after installation of P4

    psolinski

      We recently experienced many performance issues with our IT infrastructure:

      -NAS NetApp load raised 3x above average

      -file copy operation became much slower - both between servers and workstation<>server

      -Citrix logons were couple of times slower than before, user drives fail to map and printers to connect in Citrix

       

       

      As all of it started around the same time we updated out VS88P2 to Patch 4, it was the main suspect, anyway will virtually all systems already on P4 and no easy return to P2 (we cant easily install VS88P2 due to another McAfee issue - expired certificate in VS88p2 installer...) we tested influence of VS just stopping its services. There was no improvment while VS is not running so we were investigating other possible issues - like network (there was a network change the same time as P4 deployment).

       

      Two days ago our serve team discovered that servers with VS completly removed performance is back to normal.

       

       

      Knowing this I just tested write speed between 2 windows2k8r2 servers without and with VS88p4 installed:

       

      in MB/min            out MB/min      VS state

      12,249.90             22,883.30            none

      12,091.90             24,038.46            none

      24,650.78             15,764.58            none

      24,350.65             15,143.87            none

      16,025.64             21,849.96            none

       

      It's between 2 virtual machines on production vmware so speed fluctuates, but is between 24-15MB/min

       

      With VS88p4 installed on one server - the one I was writing to:

       

      19,230.77             5,128.21              VSE88p4 installed

      21,367.52             5,494.51              VSE88p4 installed

      24,038.46             5,898.55              VS services stopped

       

      So write speed went down to 25% of normal speed!!!!! Even with VS services stopped.

       

      So what can do this mess when VS services are not running? Filter drivers......

      Disabled mfe*.sys and performance went back to normal:

       

      20,215.63             25,273.80            services stopped filter drivers renamed

       

      So decided to enable them one by one:

       

      20,661.16             25,974.03            mfewfpk.sys enabled

      20,449.90             16,713.09            mfewfpk.sys enabled

      20,449.90             26,338.89            mfewfpk.sys enabled

      21,367.52             6,183.02              mfehidk.sys enabled

      20,242.91             5,933.54              mfehidk.sys enabled

      22,624.43             4,610.42              mfehidk.sys enabled

      18,315.02             25,641.03            mfehidk.sys disabled all other enabled

      18,668.33             25,641.03            mfehidk.sys disabled all other enabled

      20,242.91             23,166.02            mfehidk.sys disabled in reg all other enabled

       

       

      It clearly shows that mfehidk.sys kills your system performance.

      The same happens on vm and on phisical machines fo it's not specific to VM.

      SR 4-5606236293 opened with McAfee.

       

      It's still not clear if/how P4 affects our Citrix and NetApp environment - we are curentlly reverting what we easily can to P2.

       

       

      Some other test by our Server Team:

       

      Please see below speeds achieved with different patch level of VS

      OS

       

       

       

      no VS

      P2

                -  

      P4

      P4
      On-Access Scanner disabled

      P4
      All McAfee services stopped

      Window 2008 R2

      Writing

       

      MB/s

          264.55

          240.62

      NA

           87.67

                  77.27

                  90.12

      Window 2008 R2

      Reading

       

      MB/s

          561.80

          500.00

      NA

          533.62

                438.60

                533.05

      Window 2008 R2

      Writing

       

      %

      100%

      91%

       

      33%

      29%

      34%

      Window 2008 R2

      Reading

       

      %

      100%

      89%

       

      95%

      78%

      95%

       

       

       

       

       

       

       

       

       

       

      OS

       

       

       

      no VS

      P3

      P3 On-Access Scanner disabled

      P4

      P4
      On-Access Scanner disabled

      P4
      All McAfee services stopped

      Windows 2012 R2

      Writing

       

      MB/s

          390.32

          110.72

          110.72

          126.97

                132.24

                119.85

      Windows 2012 R2

      Reading

       

      MB/s

          726.74

          258.13

          248.02

          603.86

                542.30

                463.82

      Windows 2012 R2

      Writing

       

      %

      100%

      28%

      28%

      33%

      34%

      31%

      Windows 2012 R2

      Reading

       

      %

      100%

      36%

      34%

      83%

      75%

      64%

       

       

       

      So if you experience any strange issues after P4 - it's probably P4. Dont upgrade if you still on P2!

       

       

       

      -----------------------------------------------------

      Googling for mfehidk.sys I came across:

       

      "Obviously MFEHIDK.SYS IS Related to mfehidk.sys VirusScan Enterprise from McAfee Agent,

       

      IMHO it is one of the three most problematic malware apps for win 7 & win 8 in terms of stabiity and crashing,

       

      I would replace it with almost anything else."

        • 1. Re: Serve performance degradation after installation of P4
          jesperdb

          Nearly every type of Anti Virus = Performance degration.

           

          You mention Citrix, have you looked at their Best Practices for AV exclusions?

           

          Are you excluding anything from being scanned on read/write or both?

           

          Some simple exclusions on file extensions or even folders can improve the performance you expecrience.

           

          Message was edited by: jesperdb on 4/9/14 7:48:43 AM CDT
          • 2. Re: Serve performance degradation after installation of P4
            psolinski

            Nearly every type of Anti Virus = Performance degration.\

             

            I wouldnt say that 75% of performance degradation is anything close to acceptable.....

             

             

            You mention Citrix, have you looked at their Best Practices for AV exclusions?

             

            Are you excluding anything from being scanned on read/write or both?

             

            Some simple exclusions on file extensions or even folders can improve the performance you expecrience.

             

            We have long list of exclusions - for both folders, file types and for processes. Much more than recomended by McAfee "if you have more exclusions than your fingers it is wrong".

            And with P2 and the same list of exclusions "McAfee tax" was just 10%.

            • 3. Re: Serve performance degradation after installation of P4
              djjava9

              I have dozens of large customers that upgraded to p4 with no issues.....as mentioned by others take a close look at your policy and check exclusions.....also confirm that you are not scanning inside zip files/archives.

              • 4. Re: Serve performance degradation after installation of P4
                psolinski

                Obviouslly we dont scan archives. And the file we use to test is not compressed - just generated by genfile:

                "The default data pattern for filling the generated file consists of first 256 letters of ASCII code, repeated enough times to fill the entire file."

                500MB

                 

                As I mentioned to kill our server we dont even have to start VS services - it's enought to load  mfehidk.sys 

                This clearly means that it doesnt depend on exclusions - nothing should be scanned by OAS shen VS is not running, so nothing to exclude.

                 

                 

                Ask your customers to test SMB2.1 file write operations to server with and without vs88p4, they might change their mind.

                Or better dont.

                 

                 

                Just finished another test: replaced mfehidk.sys version 15.1.0.656 (Patch 4) with version 15.0.0.515 (Patch 2) and performance is back to normal level.

                 

                Message was edited by: psolinski on 09/04/14 17:49:29 CEST
                • 5. Re: Serve performance degradation after installation of P4
                  jesperdb

                  "mfehidk.sys            Host Intrusion Detection Link Driver. This component is used for Access Protection and by the Filter Driver and Entercept (Buffer Overflow) Driver. Altitude:321300.00"

                   

                  Have you looked in to Access Protection logs to see if anything could trigger the performance decrease there?

                   

                  maybe even take a look at this from another post:

                   

                  "

                  Hello Everybody!

                   

                   

                  I usually use McAfee Profiler in this cases to know what files n process VSE scans.

                   

                  https://kb.mcafee.com/corporate/index?page=content&id=KB69683

                   

                  With this toll I Know how many times i have I/O of the process.

                  If the Process who takes long scans and this process is a valid process maybe can be valid  Exclude this process of VSE

                   

                  I hope I have helped!"

                   

                  Message was edited by: jesperdb on 4/9/14 11:53:39 AM CDT
                  • 6. Re: Serve performance degradation after installation of P4
                    petersimmons

                    Would you please call support, open a case and get them a MER? That's the best way to make sure that we get all the data we need.

                    • 7. Re: Serve performance degradation after installation of P4
                      psolinski

                      SR 4-5606236293 with Tier II now.

                       

                      MER and procmon logs already with you.

                      • 8. Re: Serve performance degradation after installation of P4
                        Pmaquoi

                        Could you post us the conclusion of your opened SR ? i would like to know the McAfee response to this issue before eventually tryning a few upgrade myself on some servers. Thanks

                        • 9. Re: Serve performance degradation after installation of P4
                          wwarren

                          The purpose of the mfehidk.sys driver is to "hook" the operating system so it gets notified of any/all activities we _might_ be interested in analyzing further.

                          For the events we are interested in, the driver hands off to a companion driver (mfeavfk for scanning, mfeapfk for access protection, mfebopk for buffer overflow), otherwise it does nothing further but to pass the I/O on to the next entity in line.

                           

                          In saying you experience a hit with only the mfehidk.sys driver, it suggests a couple things -

                          - there might be a bug in the driver

                          - there might be an interop issue playing out with another driver(s) in your environment

                           

                          The latter can be confirmed/eliminated by reproduction attempts in a new/clean environment. An obvious one would be not using VMWare; they have drivers too you know; and that can cause issues (See KB79260 for example). Support can help work through the possibilities; reviewing MER data from a system will help so we can identify what other drivers are indeed present.

                          But if we can reproduce the issue ourselves, or it's reproducible in a clean environment then it smells like a bug.

                           

                          It's one thing to incur performance overhead due to AV software being installed, but to cut throughput by 75%? That isn't right. Throwing exclusions or performance tweaks at metrics like that is like using a bucket to save the Titanic. So, something is definitely "off" here. And because we know plenty of others are using P4 without performance woes, I'm inclined to believe there's something environmental that hasn't been identified.

                          1 2 Previous Next