9 Replies Latest reply: Mar 14, 2013 9:30 AM by brentil RSS

    VSE 8.8i Performance Tests & Results

    brentil

      We've been running VSE 8.8i on select machines since it was released and recently migrated all of our ePO controlled machines to use it.  Before doing so I performed performance tests similar to those I've done with VSE 8.5 & 8.7 in my other thread.

       

      https://community.mcafee.com/message/167738

       

      Since McAfee is claiming significant performance improvements I wanted to validate these.

      https://kc.mcafee.com/resources/sites/MCAFEE/content/live/PRODUCT_DOCUMENTATION/ 22000/PD22973/en_US/VSE%208.8%20-%20What%27s%20New.pdf

       

      Testing was the same as with my last VSE 8.7 P3 vs P4 but with 8.7 P4 vs 8.8 instead.  Results are the average of multiple tests.  The values below show the % faster VSE 8.8 is over VSE 8.7 P4.

       

       

      VSE 8.7i P4 vs VSE 8.8

       

      VSE87P4_vs_VSE88.PNG

       

      In terms of real numbers it breaks down as follows;

      • Copy
        • VSE 8.7 P4 - 27 minutes
        • VSE 8.8 - 24 minutes
      • RAR
        • VSE 8.7 P4 - 36 minutes
        • VSE 8.8 - 33 minutes
      • unRAR
        • VSE 8.7 P4 - 18 minutes
        • VSE 8.8 - 20 minutes
      • Eclipse *
        • VSE 8.7 P4 - 10.5 seconds
        • VSE 8.8 - 9 seconds
      • Oracle SQL Developer *
        • VSE 8.7 P4 - 9.5 seconds
        • VSE 8.8 - 7 seconds
      • System Scan
        • VSE 8.7 P4 - 74 minutes
        • VSE 8.8 - 12 minutes (ignoring first system scan to build cache)

       

      * - These results negate the initial scan VSE 8.8 requires to build the file list off of.  Comparisons of that information is below.

       

      VSE vs Java

       

      Probbaly the last frontier of anti-virus performance comes in relation to Java applications.  The nature of large numbers of in essence compressed files (jar files) being loaded at runtime every time results in an insane performance hit to most Java based applications.  Once they're loaded they typically behave better, but any time you intiailly load or it's cached information is lost results in massive performance hits.

       

      I'm very glad to see that VSE 8.8 helps with this some.  It's still not perfect but the margin of difference is starting to close for the first time.

       

      In the past VSE would cause a Java application to crawl to load over a period of minutes.  You can see below the Oracle SQL Developer for example would take 180 seconds 3 full minutes! to load.  After that as long as you didn't restart your machine or your VSE services cycle (like from a DAT update in some cases) you would then see a startup time in the 10 second range like other normal applications.  However once those services got cycled you were back to square one and long wait times.  It seems VSE 8.8 takes a bit longer for that very first run but subsequent runs take on average 50 seconds for the first start thanks to the persistent file caching of VSE 8.8.

      VSE87P4_vs_VSE88_Java.PNG

      • 1st - This is the first time the application is run after installing the application, no VSE cache exists for it.
      • Restart - This is the time an application takes after cycling the VSE services or rebooting the computer.
      • > 1st - This is the time it takes after the 1st or Restart time.

       

       

      As a whole VSE 8.8 seems to be a very impressive release.  We've noticed increased responsiveness of applications of various types, slower machines are less impacted (especially after the first full system scan), Full scan jobs are more often to actually complete and can also be run more often due to their faster completion, decreased memory usage overall from idle usage to active scanning.

        • 1. Re: VSE 8.8i Performance Tests & Results
          pato

          Hi

          Nice testing!

          One question, could you also add the 'System Scan' time for the first scan after a reboot? Because I think this is what most people do (shutdown the PC over the night, make daily DAT updates).

           

          Thanks,

          pato

          • 2. Re: VSE 8.8i Performance Tests & Results
            brentil

            This set of data also shows the performance benefit of VSE 8.8 over previous versions.

             

             

            VSE vs System Scans

            VSE87P4_vs_VSE88_SystemScan.PNG

             

            • 1st Scan - This is the very first System Scan performed after installing VSE
            • Restart Scan - After rebooting the computer or the VSE services cycle
            • Restart Scan + - This is any additional scan performed after a Restart Scan has been performed

             

            What I'm gathering from the data is that during the normal install/booting/running OAS scanning it's built up a decent list of files already scanned and validated.  Therefore by the time you even run your first System Scan portions of the system have already been placed in the new global cache allowing for reduced scan times.  After that thanks to the global cache any further scans are significantly improved over previous versions of VSE.

             

            If anyone else has scan ideas or information they wish to see feel free to ask!

             

             

            Message was edited by: brentil on 2/2/11 10:40:10 AM GMT-05:00
            • 3. Re: VSE 8.8i Performance Tests & Results
              sameer172006

              Good stuff Brentil.


              Just out of curiosity, do you also do any kind of malware testing/detection testing to check the efficacy of the product ?


              Sameer

              • 4. Re: VSE 8.8i Performance Tests & Results
                brentil

                The actual detection process is not my forte so I do not have any test processes in place for evaluating the effectiveness of detections.  From all the documentation it doesn't sound like the scanning process itself has changed much, just the use of the persistent file scanning cache.  I wouldn't be surprised if they computed an MD5 hash of a file when it's scanned and marked as clean thenignore the file if the hash did not change.

                • 5. 5500 Engine Benchmarking
                  brentil

                  As I've done with the last several releases of VSE I've behcnmarked the 5500 Engine Beta on my McAfee benchmark testbed.  If you're interested in my previous findings you can find the threads in my sig below.

                   

                  All tests are performed a minimum or 4 times.  The first test is thrown out and then the next 3 tests are averaged together.  If any radical results that fall wildly high or low are found they are thrown out and retested.  The machine is not connected to an ePO server and is only connected to the internet to perform an initial system update and DAT update prior to testing which at that point it is removed from the internet until testing is complete.  All tests are a series of custom scripts I've developed over the years to specifically test VSE and its updates.

                   

                  VSE88_vs_VSE88SP1_vs_VSE88SP15500.png

                   

                  My testing shows good results with the new 5500 beta engine bringing Java program execution back in line with pre SP1 results.  System Scanning however still shows a rather large increase in scanning time over pre SP1 numbers.

                  • 6. Re: 5500 Engine Benchmarking
                    amjad

                    nice effort brentil System scan is taking larger scanning time so i guess it can be improved further.

                    • 7. Re: 5500 Engine Benchmarking
                      casscoss

                      This is fantastic! Excellent job. Actual test and results from end users/supporters. Priceless in my opinion... seeing as many of the published reports from vendors are quite reputable.

                      • 8. Re: 5500 Engine Benchmarking
                        wayneb

                        brentil, very nice info.

                         

                        What tools did you use to compare the data? i'm sure my boss will ask when i upgrade to 8.8 from 8.7i P4

                        • 9. Re: 5500 Engine Benchmarking
                          brentil

                          I created my own tests to help provide metrics that would relate to my work enviroment which I think reflect general system usage as well.

                           

                          HP 6005 Pro AMD Phenom II X4 B95 (3 Ghz) with 8GB of RAM
                          W8 RTM 64bit fully patched
                          VSE 8.8i Patch 3
                          Office 2013 fully patched, Visio 2013 fully patched, Project 2013 fully patched, VS2012  fully patched, SQL2012 Management Suite fully patched
                          FireFox, Chrome, Flash, Java, TortoiseSVN, Eclipse, Oracle SQL Develope, Adobe Acrobat XI

                           

                          I've created a copy of the Program Files & Program Files (x86) folders post installation resulting in >65,000 files and 10 GB of data.  This data is used for the tests done.  Each test is run 3 times and averaged together.

                           

                          Copy - An xcopy command is used to make a complete duplicate of the files into another folder

                          RAR - The command line WinRAR application is used to make a default compression RAR file of the files

                          unRAR - The command line WinRAR is used to decompress the RAR file

                          Eclipse - The Windows Eclipse executable is loaded and timed via stop watch

                          Oracle - The Windows Oracle SQL Developer executable is loaded and timed via stop watch

                           

                          I'm actually in process of doing the same testing on the new 5600 beta2 engine as well and will be posting results shortly in that forum.

                          https://community.mcafee.com/groups/anti-malware-engine-beta?view=discussions