cancel
Showing results for 
Search instead for 
Did you mean: 
McAfee Employee

How To Implement Domain Based Rule Criteria for URL Filtering

So what if you want to allow or block an entire domain with a single entry a wildcard list?

There are a couple of use cases to consider. The first is when you want to match URL.Host only on www.domain.com and domain.com. We'll ignore the fact that www.domain.com and domain.com won't always resolve to the same server (usually they will, and in most cases administrators will want to allow (or block)domain.com if they are going to allow (or block) www.domain.com). The other use case is when you want to match domain.com, www.domain.com, foo.domain.com and foo.bar.domain.com. In short, the second use case is where you want to cover *.domain.com and domain.com with a single entry.

AFAIK the best (simplest) single entry wildcard list form for usecase 1 is: regex((www\.)?domain\.com)

AFAIK the best (simplest) single entry wildcard list form for usecase 2 is: regex((.+\.)*domain\.com)

Now a lot of people aren't comfortable with regex and like GLOB a lot better (because it's easier for the non-regex savy people to read, and it's also easier to convert existing domain lists), or they might be looking for a solution that doesn't even use wildcard lists (match operations will be faster). Here is an example ruleset that has rules that use GLOB in wildcard lists, or regex in wildcard lists, or just straight string lists to accomplish the same thing.

Rule Sets
Domain Criteria Ruleset
[This ruleset demonstrates examples of 1) Blocking an entire domain, example.com, which should cover not only www.example.com and a.b.example.com, but also example.com, without multiple entries in a wildcard list. Instead, entries are of the GLOB form *.example.com 2) Blocking an entire domain, say example.com, which should cover not only www.example.com and a.b.example.com, but also example.com, without multiple entries in the list. Instead entries are of the form: regex((.+\.)*example\.com) 3) Blocking example.com, and www.example.com with a single entry of the form: example.com in a regular string list.]
Enabled
Applies to Requests: True / Responses: True / Embedded Objects: True
Always
EnabledRuleActionEventsComments
EnabledBlock Access to All URLs With Hostnames That Match Blocked Regex Wildcard Domains
1: URL.Host matches in list Blocked Regex Wildcard Domains
Block<URL Blocked>This rule is used with lists that include entries of the form regex((.+\.)*example\.com) The rule is set up to block requests for any URLs that have hostnames that match the domain or subdomain without requiring multiple entries. This rule allows the single entry to match a hostname that is of the simple form example.com, www.example.com or a.b.c.example.com
DisabledBlock Access to All URLs With Hostnames That Match Blocked GLOB Wildcard Domains
1: URL.Host matches in list Blocked GLOB Wildcard Domains
2: OR (URL.Host does not match www.*
3: AND String.Concat("www.",URL.Host) matches in list Blocked GLOB Wildcard Domains)
Block<URL Blocked>This rule intended to be used with lists that include entries of the form *.domain.com The rule is set up to block requests for any URLs that have hostnames that match the domain or subdomain listed after the *. without requiring multiple entries. This rule allows the single *.domain.com entry to also cover a hostname that is of the simple form domain.com
DisabledBlock Access to All URLs That Match Blocked WWW Hosts List
1: URL.Host is in list Blocked WWW Hosts List
2: OR (URL.Host does not match www.*
3: AND String.Concat("www.",URL.Host) is in list Blocked WWW Hosts List)
Block<URL Blocked>This rule intended to be used with lists that include www. in the list entries. The rule is set up to block requests for URLs that have hostnames of the form www.example.com and example.com without requiring multiple entries.


Lists
String
#Blocked WWW Hosts ListHosts should include leading www and be entered in form www.example.com and www.example.co.uk
StringComment
1www.playboy.com
Wildcard Expression
#Blocked GLOB Wildcard Domains
Wildcard ExpressionComment
1*.playboy.com
#Blocked Regex Wildcard Domains
Wildcard ExpressionComment
1regex((.+\.)*playboy\.com)


Lastly there is a feature request in to create a property that enumerates all the domains and subdomains of a given host. The Set UDP.DomainList ruleset accomplishes this and puts the possible domains in a predictable order.

Rule Sets
Set UDP-DomainList_v4
[This ruleset fills string list User-Defined.DomainList with up to five possible domain entries with 1,2,3,4 and 5 labels, based on URL.Host. For example the URL.Host www.billy.joe.jim.bob.co.uk would result in a list containing joe.jim.bob.co.uk, jim.bob.co.uk bob.co.uk, co.uk, uk. The ruleset is useful for is in list criteria that match against a list of domains (or hosts with less than 6 labels in their FQDNs). Using this ruleset allows you to avoid wildcard lists and double entries. For example, if you want to match *.example.com and example.com, you would simply put example.com in the list. If you also want to add entries that match only www.example.com and example.com, you would enable the "Add Implied WWW Kludge Rule" and simply put www.example.com in the list you'll be matching against. Note that the use of this ruleset results in shorter lists without wildcards, but more matching operations. Impact on performance has not been studied. Also note that if you ONLY want to mimic the behavior of having string list with entries for both www.example.com and example.com, with half the number of entries. Then there is a much simpler way to do that. ]
Enabled
Applies to Requests: True / Responses: True / Embedded Objects: True
Always
EnabledRuleActionEventsComments
EnabledIf Hostname is IP Set UDP-DomainList to HostIsIP UDP-DomainList
1: URL.HostIsIP equals true
Stop Rule SetSet User-Defined.DomainList = HostIsIP UDP-DomainList
EnabledIf Hostname is Shortname Set UDP-DomainList to HostIsShort UDP-DomainList
1: URL.Host does not match *.*
Stop Rule SetSet User-Defined.DomainList = HostIsShort UDP-DomainList
EnabledAdd Top Level Domain to UDP-DomainList
Always
ContinueSet User-Defined.DomainList = Empty UDP-DomainList°
Set User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=[\w\-]+$)),"")
Set User-Defined.DomainList = List.OfString.Insert(User-Defined.DomainList,User-Defined.ModHostname,0)
EnabledAdd 1st Level SubDomain to UDP-DomainList
Always
ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=[\w\-]*\.[\w\-]+$)),"")
Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
EnabledAdd 2nd Level SubDomain to UDP-DomainList
1: User-Defined.ModHostname does not equal URL.Host
ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=([\w\-]*\.){2}[\w\-]+$)),"")
Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
EnabledAdd 3rd Level SubDomain to UDP-DomainList
1: User-Defined.ModHostname does not equal URL.Host
ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=([\w\-]*\.){3}[\w\-]+$)),"")
Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
EnabledAdd 4th Level SubDomain to UDP-DomainList
1: User-Defined.ModHostname does not equal URL.Host
ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=([\w\-]*\.){4}[\w\-]+$)),"")
Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
DisabledAdd implied WWW Kludge
1: URL.Host does not match www.*
ContinueSet User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)This rule is off by default, but will add "www." to URL.Host and add it at the end of User-Defined.DomainList. Useful if you want to match mcafee.com and www.mcafee.com to a list that only contains www.mcafee.com


Lists
String
#Default UDP-DomainList
StringComment
1No Entries Set
#Empty UDP-DomainList
StringComment
#HostIsIP UDP-DomainList
StringComment
1Unknown - Host Is IP
#HostIsShort UDP-DomainList
StringComment
1company.local


This User-Defined.DomainList (list of string) property can then be used to match against string lists with entries of the form domain.com, xxx, co.uk, bbc.co.uk, etc. For efficiency sake though, I would highly recommend matching a single entry in the DomainList against a single string list and separate the lists you are matching against by TLD (top level domain), 1st level sub domain etc. In other words, you would use ListOfString.Get(User-Defined.DomainList,1) is in list Blocked 1st Level SubDomain List and all entries in the Blocked 1st Level SubDomain list would be of "one dot" form (co.uk, domain.com, example.com, mcafee.com) etc.

Message was edited by: jebeling on 10/20/11 11:15:31 AM CDT
0 Kudos
14 Replies
McAfee Employee

Re: How To Implement Domain Based Rule Criteria for URL Filtering

This ruleset is now obsolete. The new URL.HostBelongsToDomains(String List) property handles all of this for you.

0 Kudos
McAfee Employee

Re: How To Implement Domain Based Rule Criteria for URL Filtering

And also URL.Domain in 7.4!

cryptochrome
Level 7

Re: How To Implement Domain Based Rule Criteria for URL Filtering

Hi Jon, what's the difference between URL.HostBelongsToDomain and the new URL.Domain?

0 Kudos
McAfee Employee

Re: How To Implement Domain Based Rule Criteria for URL Filtering

URL.Domain is a string property which contains the top level domain of the requested URL. Whereas URL.HostBelongsToDomains, is a boolean property which requires a list for it's settings, it returns true if the URL's top level domain is in the list...

2013-10-04_095834.png 2013-10-04_095859.png

(this will be in an update coming to my URL property guide https://community.mcafee.com/docs/DOC-4514)

Best,

Jon

0 Kudos
cryptochrome
Level 7

Re: How To Implement Domain Based Rule Criteria for URL Filtering

Thanks. Excuse my newbie ignorance please, but I still don't see the difference. Both of them seem to get me to the same result...

0 Kudos
McAfee Employee

Re: How To Implement Domain Based Rule Criteria for URL Filtering

True, but you cant log a boolean very nicley.

0 Kudos
cryptochrome
Level 7

Re: How To Implement Domain Based Rule Criteria for URL Filtering

I see I still have *a lot* to learn. What has that to do with logging? Let's assume I write the default access.log - would I see different results in the log when using one vs. the other property?

0 Kudos
McAfee Employee

Re: How To Implement Domain Based Rule Criteria for URL Filtering

As stated URL.HostBelongsToDomains, is a boolean property and URL.Domain is a string property.

If you wanted to log the "domain" of the url like "mcafee.com", instead of mail.mcafee.com, www.mcafee.com, download.mcafee.com etc... URL.Domain would help accomplish this.

URL.HostBelongsToDomain, would simply equal "true" or "false" instead if "mcafee.com".

Best,

Jon

0 Kudos
cryptochrome
Level 7

Re: How To Implement Domain Based Rule Criteria for URL Filtering

I understand that, but I still don't get the logging part. If using URL.HostBelongsToDomain I would have true/false in my access log instead of mcafee.com?

0 Kudos