7 Replies Latest reply on Jan 31, 2013 9:52 AM by alexott

    How long does it take your MWG to scan this Eclipse download?!   (zomg slow)

    Regis

      So this 150MB Eclipse download (very popular IDE among the Java developer community) ... really takes over 11 minutes to scan on a  1-year old 5000 series appliance running mwg 7.2.0.1 that has a load average under 1.6?         Do I have something configured very incorrectly, or is this typical?

       

      http://mirror.cc.vt.edu/pub/eclipse/technology/epp/downloads/release/juno/SR1/ec lipse-java-juno-SR1-win32-x86_64.zip

       

      This client is already skipping the composite opener at body.nestedarchive.levels at  5 or greater.

        • 1. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
          alexott

          This file contains ~91000 embedded objects, mostly the .class files, so this can cause long filtering time.

          On the first level it contains ~1700 embedded objects, so maybe you can combine the Body.NumberOfChildren property with NestedLevel?

          1 of 1 people found this helpful
          • 2. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
            Regis

            Hi Alex,

             

            This sounds very helpful.

             

            Can you say more about Body.NumberofChildren and what you're thinking here?   e.g.   OR-ing the current stopruleset  bypass rule with, say,   Body.NumberofChildren greater than ... 1000 ?    

             

            Where do we mortals find the definitions of these anyway, and/or how do we gather these sorts of metrics to do such analysis ourselves?

             

            Thanks again for your input.

            • 3. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
              alexott

              Hi

               

              The Body.NumberofChildren property should contain number of children in given archive (or document). It will be filled if archive supports analysis without complete unpacking, like for .rar, .zip, and some other archives.

               

              I'm not sure that you should use the 'Stop ruleset' here, you can just not enable openers for big archives, so rule can look like:

              if 'Body.NumberOfChildren less than 1000' AND 'Body.NestedArchiveLevel less than 5' then EnableCompositeOpeners...

               

              The precise definition of these metrics is customer-specific so I can't say what will be the best choice - it will depend on your approach to security definition, etc.

               

              Another property that you can take into account - Body.UncompressedSize - it will contain size of unpacked archive (if it's possible to calculate it without complete unpacking)...

               

              P.S. btw, in this archive it's less than 5 nested levels, so it always unpacked completely...

              1 of 1 people found this helpful
              • 4. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
                Regis

                Ah.  Even more helpful! 

                 

                So..   How doe Body.UncompressedSize   compare to Body.size?    

                 

                I ask because if  skip the composite opener and the gateway antimalware rules  if  Body.size  > 140MB  for that download,  I notice that  the very painful scanning doesn't take place and file type rules can't see inside the archive (as one might expect). 

                 

                As for our approach to security,  I'd sumarize it as never wanting a user to wait 11 minutes for the web gateway to scan 150MB (prior to virusscan on their desktop scanning that thing again.  :-).

                 

                Thanks again for your insights here.   If there are any public docs on all these properties, that'd be great -- I don't think I've been able to find property documentation anywhere, or if I have, I've forgotten where.

                • 5. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
                  alexott

                  Value of Body.UncompressedSize depends on many parameters - what files are in archive, what is compression level, etc. For given archive, the uncompressed size is around 175Mb, because there are a lot of .jar files in archive, and they are already compressed. But there are some archives, where uncompressed size can be 3-5-... times greater than original Body.Size.

                   

                  Virus scan time depends not only on size of file, but also on number of files, so one big file will be scanned faster than 1000 files of smaller size. So I think, it's better to put limit on number of files inside archive, as I described before.

                   

                  Regarding description of properties - you can find all available properties in appendix to documentation - both in PDF, and in MWG's UI - this is should be chapter "Configuration lists", section "List of Properties" (near the end of documentation)

                  • 6. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
                    Regis

                    [forehead slap]

                     

                    I read uncompressed as compressed for some reason.  Sorry for the moderately boneheaded question and the helpful answer.

                     

                    So body.uncompressedsize is simply, for a .zip or .rar file,   the size of it after the composite opener unzips or unrar's it?

                     

                    If a zip has additional compressed file types in it...     (under the nested levels limit)... will the uncompressed size be the total of al the compressed filetypes getting unzipped recursively down to that nested level?      

                     

                    I'll dig through my pdf's again -- thanks again - my understanding of the opener and how it affects things downstream is de-fogging quite a bit with this input.

                    • 7. Re: How long does it take your MWG to scan this Eclipse download?!   (zomg slow)
                      alexott

                      Body.UncompressedSize counts only uncompressed size of files that are directly stored in this archive (i.e. 1 level below current level), it doesn't go deep if there are compressed embedded objects there - otherwise it would be too slow.