I am facing longer scanning time by WG for few files for example apple.com/itunes/download.
Is the scanning duration directly proportional to the file size ? or it depends on the file format or way it has been packed?
when I tried downloading itunes from the above, it takes me almost more than 200s to scan after it downloads thru progress page.
the scanning duration cannot be related to the file size. There are 500 MB files which pass in seconds and 1 MB files that take 15 minutes 🙂
You are right, it is depending on how a file is packed and which types of files an archive (or self-extracting archive) contains. MWG takes files and extract them to the maximum level possible, or configured. So if there is an archive, which contains an archive, which contains more archive, which contains more data that we can extract (like a PDF), all of these files will be extracted and filtered. This is a huge difference to what a desktop AV does, so scanning times cannot be compared.
I have seen an MSI installer package which was 2 MB in size, but when extracting it it contained more than 9000 small files. Extracting and filtering 9000 small files certainly takes some time, so it took longer than you would expect, and also longer than the desktop AV which just looks at the 2 MB file.
So I would say iTunes is not really a very simple file, but contains a lot of data and a lot of files which we can extract and which we apply filters to. So 200 seconds should be ok.
Is there any way to log what is happening with antimalware scanning module? ... I've been experiencing same problems while e.g. downloading Firefox 13.0.1.exe from Mozilla site. It takes more than 1000 seconds to scan that executable (it has ~65 files inside of it). Device (WG 4000) CPU usage is less than 30% at that time.
there is no debug log that allows to "look into the engine". It is possible to add a log event to the composite opener rules and write down whenever an object has been extracted, but this will only tell when MWG has extracted something and does not indicate what is happening in the engine.
If the issue is reproducible it would be great if you can share the complete URL of the file you download. We can check with a default MWG and see if we can replicate the issue.
Here is an example:
Scanning time: 1100 seconds
One idea I'd throw out there is that you might consider building a trusted vendors URL list and consider it for bypass of the opener and anti-malware portions. Be careful of course of forums though to treat them a bit separately. The scan times I've seen as we've fully loaded some MWG's here lately have been eye poppingly awful, so we'll have to tune more of these. The opener level of 100 seems like one hell of a generous default, and as others have said, reducing it to 5 seems to make a difference. Finding the middle ground we can live with will be iterative.
Also, what are folks doing on a max scan size? When we were doing Bluecoat proxies at this client, a max file size to scan was 50MB. How can you specify a max size on the web gateways? Skip antimalware if response cycle Body.size > X kB?
I have downloaded the file you have mentioned above through my MWG test machine. It is a VM with 2 GB of memory and 1 CPU. The filtering took around 15 seconds.
So I think 1100 seconds are a little too long! Which version are you running on? Did you try the download with the latest 7.2 release?
We are running Server version 7, UI Version 22.214.171.124.0 (11447), but we have no planned changes for that equipment during Q3 and Q4. CPU usage is less than 30% during scanning operation. We can accept that we made errors in equipment configuration, but as this is really basic functionality that in our opinion shouldn't be dependent on software version, we can't accept need for upgrade.
Firefox is using the 7z algorithm to compress the installer. The 7z support on MWG had some glitches in the past, causing objects to take very long. I just installed a 126.96.36.199.0 build 11447 with the default configuration and I was able to replicate the issue you see, as downloading firefox took very long (I stopped after 5 minutes).
I updated this machine to the latest version and the download was completed within a couple of seconds as mentioned earlier. The only recommendation I have is to schedule an update, since this seems to solve the problem. If you have a VM or physical test machine which does not serve users you could upgrade the box and check against your configuration, I am pretty sure that the issue will vanish.
Besides whitelisting I do not see a proper way to solve the problem on 188.8.131.52.0, sorry. You could try talking to technical support to get some additional advice.