cancel
Showing results for 
Search instead for 
Did you mean: 
srini2411
Level 7

Web Gateway takes longer time to Scan files when downloaded

Hi Guys,

I am facing longer scanning time by WG for few files for example apple.com/itunes/download.

Is the scanning duration directly proportional to the file size ? or it depends on the file format or way it has been packed?

when I tried downloading itunes from the above, it takes me almost more than 200s to scan after it downloads thru progress page.

Cheers

Srini

0 Kudos
27 Replies
asabban
Level 17

Re: Web Gateway takes longer time to Scan files when downloaded

Hi Srini,

the scanning duration cannot be related to the file size. There are 500 MB files which pass in seconds and 1 MB files that take 15 minutes :-)

You are right, it is depending on how a file is packed and which types of files an archive (or self-extracting archive) contains. MWG takes files and extract them to the maximum level possible, or configured. So if there is an archive, which contains an archive, which contains more archive, which contains more data that we can extract (like a PDF), all of these files will be extracted and filtered. This is a huge difference to what a desktop AV does, so scanning times cannot be compared.

I have seen an MSI installer package which was 2 MB in size, but when extracting it it contained more than 9000 small files. Extracting and filtering 9000 small files certainly takes some time, so it took longer than you would expect, and also longer than the desktop AV which just looks at the 2 MB file.

So I would say iTunes is not really a very simple file, but contains a lot of data and a lot of files which we can extract and which we apply filters to. So 200 seconds should be ok.

Best,

Andre

0 Kudos
srini2411
Level 7

Re: Web Gateway takes longer time to Scan files when downloaded

Thanks Andre  for your valuable suggestion.

0 Kudos
msm
Level 9

Re: Web Gateway takes longer time to Scan files when downloaded

Is there any way to log what is happening with antimalware scanning module? ... I've been experiencing same problems while e.g. downloading Firefox 13.0.1.exe from Mozilla site. It takes more than 1000 seconds to scan that executable (it has ~65 files inside of it). Device (WG 4000) CPU usage is less than 30% at that time.

Thanks, MSM

0 Kudos
asabban
Level 17

Re: Web Gateway takes longer time to Scan files when downloaded

Hello,

there is no debug log that allows to "look into the engine". It is possible to add a log event to the composite opener rules and write down whenever an object has been extracted, but this will only tell when MWG has extracted something and does not indicate what is happening in the engine.

If the issue is reproducible it would be great if you can share the complete URL of the file you download. We can check with a default MWG and see if we can replicate the issue.

Best,

Andre

0 Kudos
msm
Level 9

Re: Web Gateway takes longer time to Scan files when downloaded

Hello Andre,

Here is an example:

Date: 24.7.2012

Link: http://download.cdn.mozilla.net/pub/mozilla.org/metrics/14.0.1-funnelcake14/win32/en-US/Firefox Setu...

Scanning time: 1100 seconds

Thanks, MSM

0 Kudos
Regis
Level 12

Re: Web Gateway takes longer time to Scan files when downloaded

One idea I'd throw out there is that you might consider building a trusted vendors URL list and consider it for bypass of the opener and anti-malware portions.  Be careful of course of forums though to treat them a bit separately.     The scan times I've seen as we've fully loaded some MWG's here lately have been eye poppingly awful, so we'll have to tune more of these.    The opener level of 100 seems like one hell of a generous default, and as others have said, reducing it to 5 seems to make a difference.  Finding the middle ground we can live with will be iterative.

Also, what are folks doing on a max scan size?  When we were doing Bluecoat proxies at this client, a max file size to scan was 50MB.   How can you specify a max size on the web gateways?    Skip antimalware if response cycle Body.size > X kB?  

0 Kudos
asabban
Level 17

Re: Web Gateway takes longer time to Scan files when downloaded

Hello,

I have downloaded the file you have mentioned above through my MWG test machine. It is a VM with 2 GB of memory and 1 CPU. The filtering took around 15 seconds.

So I think 1100 seconds are a little too long! Which version are you running on? Did you try the download with the latest 7.2 release?

Best,

Andre

0 Kudos
msm
Level 9

Re: Web Gateway takes longer time to Scan files when downloaded

Hello Andre

We are running Server version 7, UI Version 7.1.5.1.0 (11447), but we have no planned changes for that equipment during Q3 and Q4. CPU usage is less than 30% during scanning operation. We can accept that we made errors in equipment configuration, but as this is really basic functionality that in our opinion shouldn't be dependent on software version, we can't accept need for upgrade.

Thanks, MSM

0 Kudos
asabban
Level 17

Re: Web Gateway takes longer time to Scan files when downloaded

Hello,

Firefox is using the 7z algorithm to compress the installer. The 7z support on MWG had some glitches in the past, causing objects to take very long. I just installed a 7.1.5.1.0 build 11447 with the default configuration and I was able to replicate the issue you see, as downloading firefox took very long (I stopped after 5 minutes).

I updated this machine to the latest version and the download was completed within a couple of seconds as mentioned earlier. The only recommendation I have is to schedule an update, since this seems to solve the problem.  If you have a VM or physical test machine which does not serve users you could upgrade the box and check against your configuration, I am pretty sure that the issue will vanish.

Besides whitelisting I do not see a proper way to solve the problem on 7.1.5.1.0, sorry. You could try talking to technical support to get some additional advice.

Best,

Andre