In Web Gateway 6.x there is an option to block the referer that is normally sent in the HTTP request (Common > Privacy Filters > Referer Filter)
How would one do the same in mwg7? Specifically I would like to remove the referer if the domain is different. Yes I am aware that this can cause problems for certain web sites that deny traffic unless the correct referer is sent (crappy form of access control in my opinion). But from what I've seen removing the referer also blocks malware on certain sites that only gets delivered to you if the referer field is a search engine (google, bing, yahoo) or a redirector as opposed to a direct request to the malicious URL.
to remove the referer header you can use the Header.RemoveAll event and tell it to remove all headers with the name "referer".
To specify when to remove the header you have to find the appropriate criteria, depending on what you would like to do. You could compare the value of the URL accessed and the value in the referer header. I did this a while ago by checking whether the content of the referer header (stored in the property Header.Request.Get('Referer')) with the value of the URL that was currently accessed, which is stored in URL.Host. I remember this was working as it did in 6.x, but I think I don't have the rule set anymore.
Maybe you can have a look it you can get it setup as you desire.
Maybe I'm doing something wrong (I'm new to mwg7) however I had tried doing that but am unable to select what I would think would be the the right operator.
In the Add Criteria screen, on the left I select URL.Host. On the right I select the property Header.Request.Get('Referer'). In the middle for the operator the only one that I can select is "does not equal" instead of something that does a wildcard search (the referer field will contain the full URL whereas the URL.Host will only be the URL hostname, so I need to match www.example.com into http://www.example.com/dir/index.html). The best I can do right now is block the referer all the time whereas I only want to block it if the domain is different.
PS: Is there a function that outputs the host for any given string that contains a URL? Ex: something like URL.Host(Header.Request.Get('Referer'))?
yes it is not as simple as it sounds :-) You actually want to match a string against a wildcard expression, which are two different data types in MWG. You have "URL.Host" which is a string. You want to use "match" (not equals, because that won't ever match). To use match you have to pick a wildcard expression on the operator side. This does not work out of the box.
Additionally the Referer filter in MWG6 was domain wide, not host wide. You may want to keep the referer filter if you are forwarded from www.google.com to search.google.com.
I have added a rule set which could help, I have not really tested it.
In the first rule it takes the URL.Host property, extracts the URL via Regex and adds the result from this to a string which has an asterisk on its start and end. So I have:
and I want:
To do this I had to use a user-defined property.
In the second step I use the value of the referer header and check if it matches my wildcard (you will see that I use String.ToWildcard to convert my plain string into something I can use as a wildcard operator).
So I say:
http://www.google.de/search?q=andre&anotherparm=funny does not match *google.de*
then remove headers.
It should basically work. Maybe you want to have a look and check if it makes some sense.
Nachricht geändert durch asabban on 26.11.12 16:02:11 MEZ
Thank you. This answered my question and got me on the right track. I've modified your ruleset slightly as follows:
In yours you use a regex so that www.google.de becomes the wildcard *google.de* which you store in User-Defined.URL.Domain.
In mine, I've modified it to use a simpler string concatenation instead of your regex so that www.google.de becomes the wildcard *://www.google.de/*
I do this to avoid the scenario where the referer still gets passed along when a user clicks on a search result in google instead of being blocked, i.e.
Referer = http://www.google.com/url?<VariousParameters>&url=http%3A%2F%2www.website-being-clicked-on.com&<MoreParameters>
This will match *website-being-clicked-on.com* causing the referer field to still get passed instead of being removed.