4 Replies Latest reply on Feb 17, 2016 2:05 AM by asabban

    Download managers and Progress Indication

    msiemens

      We're encountering more and more applications that use download managers. When download managers initiate the progress page, the user just sees that their application is hung. As a result, I have to dig into the logs to see what URL is being used and add it to the progress page bypass rule.

       

      I think that I could make our users happier and save myself a lot of research time if I bypassed the progress page when the agent ID indicated something other than a browser (agent ID != Chrome|IE|Firefox|Safari) and let the download manager use data trickling. Then the download could occur and the user's application wouldn't hang.

       

      Has anyone considered this approach? Is there a ruleset that detects Chrome|IE|Firefox|Safari?

       

      Mike

        • 1. Re: Download managers and Progress Indication
          malefunk

          Hi,

           

          Add Ruleset from Library -> Common Rules -> Progress Indication

           

          Progress Page is only enabled if the User Agent Header contains *mozilla*. Additionally to the default Set we made an Rule with an URL Whitelist for sites which don't work with Progress Page, Vmware Downloads for example.

          • 2. Re: Download managers and Progress Indication
            Jon Scholten

            Hi Mike,

             

            malefunk is correct about the existing progress page rule, it should only execute if we detect a browser (aka Mozilla -- this doesnt mean only Firefox). My colleague mgrider wrote an awesome guide about progress pages and talks about what Mozilla means, as well as other progress page tips: Progress Indication Explained

             

            Another problem to consider with download managers is the fact that they dont try to download the entire file. They try to only get a piece of it, this is called partial downloads (http response code 206). The user will specify a "Range" header whereby the client will indicate what bytes of a file they want.

             

            By default, MWG doesnt allow this to happen, instead we force the user to get the entire file. If we did, then MWG wouldnt be able to scan the entire file.

             

            Under your Gateway-Antimalware rules there is a rule called "Remove Partial Content for HTTP(s) requests". This might be the rule preventing the download managers from obtaining partial files (common when people pause the download).

             

            Best Regards,

            Jon

            • 3. Re: Download managers and Progress Indication
              msiemens

              Are you saying that only browsers can have *mozilla* in the user agent? I'm not talking about partial downloads. I'm referring to applications that fetch data in the background but don't display a progress page. These are the applications that appear "hung" to a user and can be very difficult to troubleshoot without a real time session with the user.

              • 4. Re: Download managers and Progress Indication
                asabban

                Hello!

                 

                The problem with the "User-Agent" header is that it is generally a free text field where you as a piece of software utilizing HTTP can enter whatever you want. Using *mozilla* is the most simple thing to make sure you catch all major browsers, but of course also non-browser clients like command line tools, download managers, Microsofts "BITS" etc. may match as well.

                 

                For us as a gateway unfortunately it is not easy to detect whether there is a user that made the request from a browser or not (which means show progress pages or not). You could look into a list of known User-Agent headers at UserAgentString.com - List of User Agent Strings ... there is a very big amount of available User-Agents and - as mentioned - because it is a text field the list is incomplete.

                 

                I am not aware that the "*mozilla*" criteria for calling progress pages generally causes issues, so I would not say this is a "well known" problem that many customers stumble across. It seems that you are running into problems every now and then and as mentioned, troubleshooting is not easy here. It should be easy to modify the criteria to look more closely for a known browser User-Agent and use Data Trickling for all other downloads, but there are some things to consider:

                 

                - Users may find a browser that uses a User-Agent that does not match and may complain about "slow transfer rate" when using Data Trickling rather than Progress Pages

                - Some "clients" that perform a non-interactive download may simply call a system library or similar and make use of an installed browser which would cause the User-Agent to indicate a real browser

                 

                If we take some User-Agents from known browsers, they look like this:

                 

                • Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36
                • Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A
                • Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
                • Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko
                • Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246

                 

                This could be a match for all of them:

                 

                regex(^Mozilla\/[\d\.]+.*(?:Chrome|Safari|Firefox|MSIE).*$)

                 

                To analyze/modify/test the regex see here:

                 

                https://regex101.com/r/uT6gC3/1

                 

                You could try if that make the situation better for you.

                 

                Best,

                Andre