We're encountering more and more applications that use download managers. When download managers initiate the progress page, the user just sees that their application is hung. As a result, I have to dig into the logs to see what URL is being used and add it to the progress page bypass rule.
I think that I could make our users happier and save myself a lot of research time if I bypassed the progress page when the agent ID indicated something other than a browser (agent ID != Chrome|IE|Firefox|Safari) and let the download manager use data trickling. Then the download could occur and the user's application wouldn't hang.
Has anyone considered this approach? Is there a ruleset that detects Chrome|IE|Firefox|Safari?
Add Ruleset from Library -> Common Rules -> Progress Indication
Progress Page is only enabled if the User Agent Header contains *mozilla*. Additionally to the default Set we made an Rule with an URL Whitelist for sites which don't work with Progress Page, Vmware Downloads for example.
is correct about the existing progress page rule, it should only execute if we detect a browser (aka Mozilla -- this doesnt mean only Firefox). My colleague wrote an awesome guide about progress pages and talks about what Mozilla means, as well as other progress page tips:
Another problem to consider with download managers is the fact that they dont try to download the entire file. They try to only get a piece of it, this is called partial downloads (http response code 206). The user will specify a "Range" header whereby the client will indicate what bytes of a file they want.
By default, MWG doesnt allow this to happen, instead we force the user to get the entire file. If we did, then MWG wouldnt be able to scan the entire file.
Under your Gateway-Antimalware rules there is a rule called "Remove Partial Content for HTTP(s) requests". This might be the rule preventing the download managers from obtaining partial files (common when people pause the download).
Are you saying that only browsers can have *mozilla* in the user agent? I'm not talking about partial downloads. I'm referring to applications that fetch data in the background but don't display a progress page. These are the applications that appear "hung" to a user and can be very difficult to troubleshoot without a real time session with the user.
The problem with the "User-Agent" header is that it is generally a free text field where you as a piece of software utilizing HTTP can enter whatever you want. Using *mozilla* is the most simple thing to make sure you catch all major browsers, but of course also non-browser clients like command line tools, download managers, Microsofts "BITS" etc. may match as well.
For us as a gateway unfortunately it is not easy to detect whether there is a user that made the request from a browser or not (which means show progress pages or not). You could look into a list of known User-Agent headers at UserAgentString.com - List of User Agent Strings ... there is a very big amount of available User-Agents and - as mentioned - because it is a text field the list is incomplete.
I am not aware that the "*mozilla*" criteria for calling progress pages generally causes issues, so I would not say this is a "well known" problem that many customers stumble across. It seems that you are running into problems every now and then and as mentioned, troubleshooting is not easy here. It should be easy to modify the criteria to look more closely for a known browser User-Agent and use Data Trickling for all other downloads, but there are some things to consider:
- Users may find a browser that uses a User-Agent that does not match and may complain about "slow transfer rate" when using Data Trickling rather than Progress Pages
- Some "clients" that perform a non-interactive download may simply call a system library or similar and make use of an installed browser which would cause the User-Agent to indicate a real browser
If we take some User-Agents from known browsers, they look like this:
This could be a match for all of them:
To analyze/modify/test the regex see here:
You could try if that make the situation better for you.