Feb 1, 2011 2:59 PM
Leith Tussing - University of Central Florida
We've been running VSE 8.8i on select machines since it was released and recently migrated all of our ePO controlled machines to use it. Before doing so I performed performance tests similar to those I've done with VSE 8.5 & 8.7 in my other thread.
Since McAfee is claiming significant performance improvements I wanted to validate these.
Testing was the same as with my last VSE 8.7 P3 vs P4 but with 8.7 P4 vs 8.8 instead. Results are the average of multiple tests. The values below show the % faster VSE 8.8 is over VSE 8.7 P4.
In terms of real numbers it breaks down as follows;
* - These results negate the initial scan VSE 8.8 requires to build the file list off of. Comparisons of that information is below.
Probbaly the last frontier of anti-virus performance comes in relation to Java applications. The nature of large numbers of in essence compressed files (jar files) being loaded at runtime every time results in an insane performance hit to most Java based applications. Once they're loaded they typically behave better, but any time you intiailly load or it's cached information is lost results in massive performance hits.
I'm very glad to see that VSE 8.8 helps with this some. It's still not perfect but the margin of difference is starting to close for the first time.
In the past VSE would cause a Java application to crawl to load over a period of minutes. You can see below the Oracle SQL Developer for example would take 180 seconds 3 full minutes! to load. After that as long as you didn't restart your machine or your VSE services cycle (like from a DAT update in some cases) you would then see a startup time in the 10 second range like other normal applications. However once those services got cycled you were back to square one and long wait times. It seems VSE 8.8 takes a bit longer for that very first run but subsequent runs take on average 50 seconds for the first start thanks to the persistent file caching of VSE 8.8.
As a whole VSE 8.8 seems to be a very impressive release. We've noticed increased responsiveness of applications of various types, slower machines are less impacted (especially after the first full system scan), Full scan jobs are more often to actually complete and can also be run more often due to their faster completion, decreased memory usage overall from idle usage to active scanning.
One question, could you also add the 'System Scan' time for the first scan after a reboot? Because I think this is what most people do (shutdown the PC over the night, make daily DAT updates).
This set of data also shows the performance benefit of VSE 8.8 over previous versions.
What I'm gathering from the data is that during the normal install/booting/running OAS scanning it's built up a decent list of files already scanned and validated. Therefore by the time you even run your first System Scan portions of the system have already been placed in the new global cache allowing for reduced scan times. After that thanks to the global cache any further scans are significantly improved over previous versions of VSE.
If anyone else has scan ideas or information they wish to see feel free to ask!
Message was edited by: brentil on 2/2/11 10:40:10 AM GMT-05:00
The actual detection process is not my forte so I do not have any test processes in place for evaluating the effectiveness of detections. From all the documentation it doesn't sound like the scanning process itself has changed much, just the use of the persistent file scanning cache. I wouldn't be surprised if they computed an MD5 hash of a file when it's scanned and marked as clean thenignore the file if the hash did not change.
As I've done with the last several releases of VSE I've behcnmarked the 5500 Engine Beta on my McAfee benchmark testbed. If you're interested in my previous findings you can find the threads in my sig below.
All tests are performed a minimum or 4 times. The first test is thrown out and then the next 3 tests are averaged together. If any radical results that fall wildly high or low are found they are thrown out and retested. The machine is not connected to an ePO server and is only connected to the internet to perform an initial system update and DAT update prior to testing which at that point it is removed from the internet until testing is complete. All tests are a series of custom scripts I've developed over the years to specifically test VSE and its updates.
My testing shows good results with the new 5500 beta engine bringing Java program execution back in line with pre SP1 results. System Scanning however still shows a rather large increase in scanning time over pre SP1 numbers.
nice effort brentil System scan is taking larger scanning time so i guess it can be improved further.
This is fantastic! Excellent job. Actual test and results from end users/supporters. Priceless in my opinion... seeing as many of the published reports from vendors are quite reputable.
I created my own tests to help provide metrics that would relate to my work enviroment which I think reflect general system usage as well.
HP 6005 Pro AMD Phenom II X4 B95 (3 Ghz) with 8GB of RAM
W8 RTM 64bit fully patched
VSE 8.8i Patch 3
Office 2013 fully patched, Visio 2013 fully patched, Project 2013 fully patched, VS2012 fully patched, SQL2012 Management Suite fully patched
FireFox, Chrome, Flash, Java, TortoiseSVN, Eclipse, Oracle SQL Develope, Adobe Acrobat XI
I've created a copy of the Program Files & Program Files (x86) folders post installation resulting in >65,000 files and 10 GB of data. This data is used for the tests done. Each test is run 3 times and averaged together.
Copy - An xcopy command is used to make a complete duplicate of the files into another folder
RAR - The command line WinRAR application is used to make a default compression RAR file of the files
unRAR - The command line WinRAR is used to decompress the RAR file
Eclipse - The Windows Eclipse executable is loaded and timed via stop watch
Oracle - The Windows Oracle SQL Developer executable is loaded and timed via stop watch
I'm actually in process of doing the same testing on the new 5600 beta2 engine as well and will be posting results shortly in that forum.