1 2 Previous Next 14 Replies Latest reply on Oct 17, 2013 5:25 AM by roybad

    How To Implement Domain Based Rule Criteria for URL Filtering

    jebeling

      So what if you want to allow or block an entire domain with a single entry a wildcard list?

       

      There are a couple of use cases to consider. The first is when you want to match URL.Host only on www.domain.com and domain.com. We'll ignore the fact that www.domain.com and domain.com won't always resolve to the same server (usually they will, and in most cases administrators will want to allow (or block)domain.com if they are going to allow (or block) www.domain.com). The other use case is when you want to match domain.com, www.domain.com, foo.domain.com and foo.bar.domain.com. In short, the second use case is where you want to cover *.domain.com and domain.com with a single entry.

       

      AFAIK the best (simplest) single entry wildcard list form for usecase 1 is: regex((www\.)?domain\.com)

      AFAIK the best (simplest) single entry wildcard list form for usecase 2 is: regex((.+\.)*domain\.com)

       

      Now a lot of people aren't comfortable with regex and like GLOB a lot better (because it's easier for the non-regex savy people to read, and it's also easier to convert existing domain lists), or they might be looking for a solution that doesn't even use wildcard lists (match operations will be faster). Here is an example ruleset that has rules that use GLOB in wildcard lists, or regex in wildcard lists, or just straight string lists to accomplish the same thing.

       

      Rule Sets
      Domain Criteria Ruleset
      [This ruleset demonstrates examples of 1) Blocking an entire domain, example.com, which should cover not only www.example.com and a.b.example.com, but also example.com, without multiple entries in a wildcard list. Instead, entries are of the GLOB form *.example.com 2) Blocking an entire domain, say example.com, which should cover not only www.example.com and a.b.example.com, but also example.com, without multiple entries in the list. Instead entries are of the form: regex((.+\.)*example\.com) 3) Blocking example.com, and www.example.com with a single entry of the form: example.com in a regular string list.]
      Enabled
      Applies to Requests: True / Responses: True / Embedded Objects: True
      Always
      EnabledRuleActionEventsComments
      EnabledBlock Access to All URLs With Hostnames That Match Blocked Regex Wildcard Domains
      1: URL.Host matches in list Blocked Regex Wildcard Domains
      Block<URL Blocked>This rule is used with lists that include entries of the form regex((.+\.)*example\.com) The rule is set up to block requests for any URLs that have hostnames that match the domain or subdomain without requiring multiple entries. This rule allows the single entry to match a hostname that is of the simple form example.com, www.example.com or a.b.c.example.com
      DisabledBlock Access to All URLs With Hostnames That Match Blocked GLOB Wildcard Domains
      1: URL.Host matches in list Blocked GLOB Wildcard Domains
      2: OR (URL.Host does not match www.*
      3: AND String.Concat("www.",URL.Host) matches in list Blocked GLOB Wildcard Domains)
      Block<URL Blocked>This rule intended to be used with lists that include entries of the form *.domain.com The rule is set up to block requests for any URLs that have hostnames that match the domain or subdomain listed after the *. without requiring multiple entries. This rule allows the single *.domain.com entry to also cover a hostname that is of the simple form domain.com
      DisabledBlock Access to All URLs That Match Blocked WWW Hosts List
      1: URL.Host is in list Blocked WWW Hosts List
      2: OR (URL.Host does not match www.*
      3: AND String.Concat("www.",URL.Host) is in list Blocked WWW Hosts List)
      Block<URL Blocked>This rule intended to be used with lists that include www. in the list entries. The rule is set up to block requests for URLs that have hostnames of the form www.example.com and example.com without requiring multiple entries.


       

      Lists
      String
      #Blocked WWW Hosts ListHosts should include leading www and be entered in form www.example.com and www.example.co.uk
      StringComment
      1www.playboy.com
      Wildcard Expression
      #Blocked GLOB Wildcard Domains
      Wildcard ExpressionComment
      1*.playboy.com
      #Blocked Regex Wildcard Domains
      Wildcard ExpressionComment
      1regex((.+\.)*playboy\.com)


       

      Lastly there is a feature request in to create a property that enumerates all the domains and subdomains of a given host. The Set UDP.DomainList ruleset accomplishes this and puts the possible domains in a predictable order.

       

      Rule Sets
      Set UDP-DomainList_v4
      [This ruleset fills string list User-Defined.DomainList with up to five possible domain entries with 1,2,3,4 and 5 labels, based on URL.Host. For example the URL.Host www.billy.joe.jim.bob.co.uk would result in a list containing joe.jim.bob.co.uk, jim.bob.co.uk bob.co.uk, co.uk, uk. The ruleset is useful for is in list criteria that match against a list of domains (or hosts with less than 6 labels in their FQDNs). Using this ruleset allows you to avoid wildcard lists and double entries. For example, if you want to match *.example.com and example.com, you would simply put example.com in the list. If you also want to add entries that match only www.example.com and example.com, you would enable the "Add Implied WWW Kludge Rule" and simply put www.example.com in the list you'll be matching against. Note that the use of this ruleset results in shorter lists without wildcards, but more matching operations. Impact on performance has not been studied. Also note that if you ONLY want to mimic the behavior of having string list with entries for both www.example.com and example.com, with half the number of entries. Then there is a much simpler way to do that. ]
      Enabled
      Applies to Requests: True / Responses: True / Embedded Objects: True
      Always
      EnabledRuleActionEventsComments
      EnabledIf Hostname is IP Set UDP-DomainList to HostIsIP UDP-DomainList
      1: URL.HostIsIP equals true
      Stop Rule SetSet User-Defined.DomainList = HostIsIP UDP-DomainList
      EnabledIf Hostname is Shortname Set UDP-DomainList to HostIsShort UDP-DomainList
      1: URL.Host does not match *.*
      Stop Rule SetSet User-Defined.DomainList = HostIsShort UDP-DomainList
      EnabledAdd Top Level Domain to UDP-DomainList
      Always
      ContinueSet User-Defined.DomainList = Empty UDP-DomainList°
      Set User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=[\w\-]+$)),"")
      Set User-Defined.DomainList = List.OfString.Insert(User-Defined.DomainList,User-Defined.ModHostname,0)
      EnabledAdd 1st Level SubDomain to UDP-DomainList
      Always
      ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=[\w\-]*\.[\w\-]+$)),"")
      Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
      EnabledAdd 2nd Level SubDomain to UDP-DomainList
      1: User-Defined.ModHostname does not equal URL.Host
      ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=([\w\-]*\.){2}[\w\-]+$)),"")
      Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
      EnabledAdd 3rd Level SubDomain to UDP-DomainList
      1: User-Defined.ModHostname does not equal URL.Host
      ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=([\w\-]*\.){3}[\w\-]+$)),"")
      Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
      EnabledAdd 4th Level SubDomain to UDP-DomainList
      1: User-Defined.ModHostname does not equal URL.Host
      ContinueSet User-Defined.ModHostname = String.ReplaceFirstMatch(URL.Host,regex(^.*\.(?=([\w\-]*\.){4}[\w\-]+$)),"")
      Set User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)
      DisabledAdd implied WWW Kludge
      1: URL.Host does not match www.*
      ContinueSet User-Defined.DomainList = List.OfString.Append(User-Defined.DomainList,User-Defined.ModHostname)This rule is off by default, but will add "www." to URL.Host and add it at the end of User-Defined.DomainList. Useful if you want to match mcafee.com and www.mcafee.com to a list that only contains www.mcafee.com


       

      Lists
      String
      #Default UDP-DomainList
      StringComment
      1No Entries Set
      #Empty UDP-DomainList
      StringComment
      #HostIsIP UDP-DomainList
      StringComment
      1Unknown - Host Is IP
      #HostIsShort UDP-DomainList
      StringComment
      1company.local


       

      This User-Defined.DomainList (list of string) property can then be used to match against string lists with entries of the form domain.com, xxx, co.uk, bbc.co.uk, etc. For efficiency sake though, I would highly recommend matching a single entry in the DomainList against a single string list and separate the lists you are matching against by TLD (top level domain), 1st level sub domain etc. In other words, you would use ListOfString.Get(User-Defined.DomainList,1) is in list Blocked 1st Level SubDomain List and all entries in the Blocked 1st Level SubDomain list would be of "one dot" form (co.uk, domain.com, example.com, mcafee.com) etc.

       

      Message was edited by: jebeling on 10/20/11 11:15:31 AM CDT
        1 2 Previous Next