1 of 1 people found this helpful
The 403 and 404 are comming from the webserver and not the proxy.
An example 403 on WW7 looks like this
[root@server ~]# curl -x 10.134.144.99:80 playboy.com -I
HTTP/1.1 403 URLBlocked
Via: 1.1 10.134.144.99 (McAfee Web Gateway 188.8.131.5281)
Note it returns a block category in the 403 message.
I think there is an issue with the webserver, as though it works when accessing with out a proxy server, it does not work when trying webwasher 6, 7.1, 7.2 or bluecoat sg 5.5 and it times out when trying curl direct to the server.
Thank you for the feedback. Having the results with Bluecoat and Webwasher versions it's good information on my troubleshooting. Althougt while working with the direct access is a hard fact to my customer.
Again, thank you very much for the info.
the Server does not like the "X-Forwarded-For" header MWG adds. Use a rule as below to solve the problem:
PS: This one should work as well: https://contentsecurity.mcafee.com/ruleset_library?q=50011
This did solve the problem.
How did you realize that this was the problem? Is there any information in logs that point to the correct solution or it was related to your expertise in HTTP communications? I'm asking this because I didn't find anything that would point to this solution.
Thank you very much for your help, it was very very helpfull.
1 of 1 people found this helpful
It was an assumption :-) Basically what I did was the following:
- Copied the URL
- Browsed it with a browser which has MWG as a proxy enabled -> Saw the 403
- Browsed it with the same browser but without MWG -> Page worked
Because the response I saw when browsing through MWG was a message that was generated at the server (not by MWG) I knew that the server responds in a different way when there is MWG in the loop. If you add a proxy, usually the only difference for the server is maybe the source IP address (even this is not true when there is a NAT in place (which usually is)), and the request headers. The only headers MWG adds by default are Via and X-Forwarded-For. Usually on the last proxy within your company you should remove the X-Forwarded-For header, as it contains information about the proxies you have passed, and this becomes available to the web server. But many customers do not do so and I have already seen one or two web servers acting really weird when there are additional headers they do not expect.
So I tried to remove both headers and browsed through MWG again. The page worked fine so I removed the removal of the Via Header. The page still worked. Then I removed the rule that removes the X-Forwarded-For header and refreshed and I encountered the 403 again.
The X-Forwarded-For header itself is find and there is no problem with having it enabled. That the server sends such a response is something that goes wrong on the server side (maybe a very strict IPS system or a broken/malconfigured web server). For MWG all is perfect. The request the browser sends is fine, the request MWG sends to the server is fine and even the response is perfectly fine. MWG does not know that there *should* not be a 403, but content coming back. For MWG this means the user has requested something the web server does not want to present, but the forbidden message is basically an acceptable response.
Therefore it is hard or almost impossible to have such a behavior trigger an alarm that could help an administrator to troubleshoot. There are many reasons for 4xx status codes on the web server, and I would say 99% are perfectly fine, so you would have a lot of false alerts :-)
It is always good to think about "what does the proxy change" compared to an access without the proxy. I have seen probably a dozen of occurences where (on the remote end) servers start acting weird if they do not get the request they expect. It seems that many servers and web applications do not carefully test with a browser in the loop. Imagine you build a web site... you access it with a browser and if it works - fine. You do not think or care about proxies in the loop, so I can understand this.
I have seen the following:
- Server delivering wrong response (your case)
- Server not responding at all, but keeping the connection open until tcp timeout kicks in (a while ago in the community)
- Application running on server not sending any data causing MWG to time out before application finishes (application should send some bytes to keep connection alive)
- Web application sending the same response in an endless loop (odd!)
- Application stripping content from the response
This kind of "weirdness" is hard to find, but in many cases it is not a wrong behaviour on MWG and we can find a workaround.
I hope this little "overview" helps you to understand what happens. And as usual support and community will assist in such cases :-)
This overview was very helpfull, really. Very well explained,too.
I know I can count on support and community but I do a lot's of troubleshooting (this and other Mcafee products) and knowing how things are solved, is gold to me.
you are welcome :-) I am happy to hear that.
Is it possible to create a list of specific URL's or domains and add the Remove Proxy headers to only apply to the list?
If you follow the rule that has been shown above simply change the criteria from "Always" to "URL" matches in list XYZ or "URL.Host" matches in list ABC and create a list with the URLs you would like to apply the rule to. Pretty simple :-)