Skip navigation
McAfee Secure sites help keep you safe from identity theft, credit card fraud, spyware, spam, viruses and online scams
Currently Being Moderated

Best Practices: Creating URL related list entries

VERSION 4  Click to view document history
Created on: Jan 3, 2013 5:01 PM by Jon Scholten - Last Modified:  Oct 4, 2013 1:12 PM by Jon Scholten

 

Introduction

I have written the following guide to help understand the use of properties within rules, and how to formulate list entries to go with the corresponding rules. For example, a common question I get a lot is... I added [INSERT-SITE-HERE].com to [INSERT-LIST-HERE], but the site is still blocked. Understanding the rule criteria is essential in managing the Web Gateway's rules and how they apply. This article will attempt to simplify some very common examples and explain use cases of certain properties. To start I will be focusing on URL based properties only.

 

 

Best Practices

If you read any piece of this document, please at least read this section. After you read this, you can use the "Good/Bad Examples" for further detail and reference. The below examples outline use cases for the most commonly used URL related properties.

 

Whitelisting an entire domain (use URL.Domain or URL.Host.BelongsToDomains)

Goal: Allow all of 'mcafee.com'.

Action: Create or find a rule that uses the property 'URL.Domain' (for 7.4+) or 'URL.Host.BelongsToDomains' and define 'mcafee.com' in the list used by it.

What it resembles:

 

1.0.0v2_urldomain_whitelist.png

 

1.0.0_belongs_whitelist.png

 

Whitelisting a specific host, no subdomains (use URL.Host)

 

Goal: Allow only 'download.mcafee.com', NOT 'mcafee.com' or 'secure1.download.mcafee.com'.

Action: Create or find a rule that uses the property 'URL.Host' and define 'download.mcafee.com' in the list used by the rule.

What it resembles:

 

1.0.1_urlhost_whitelist.png

 

Whitelisting a specific download (use URL)

Goal: Allow users to download a specific file, in this case 'hxxp://download.mcafee.com/products/mcafee-avert/Stinger/Stinger.exe'.

Action: Create or find a rule that uses the property 'URL'

What it resembles:

 

1.0.2_fullurl_whitelist.png

 

 

Example URL Breakdown

Example URL

http://www.mcafee.com/us/products/web-gateway.aspx

 

URL

http://www.mcafee.com/us/products/web-gateway.aspx

 

URL.Host

www.mcafee.com

 

URL.Domain (7.4+)

mcafee.com

 

URL.Host.BelongsToDomains (example entry)

mcafee.com

 

URL.Protocol

http

 

URL.Path

/us/products/web-gateway.aspx

 

Operator importance

is in list

Use of "is in list" implies exact string match. Wildcard characters will be interpretted as literal strings.

 

matches in list

Use of "matches in list" allows for wildcard matches. Although wildcard characters are accepted, they are not completely necessary (as shown in the example for 'Whitelisting a specific host, no subdomains (use URL.Host)' above).

 

 

Good/Bad Examples by Property

The following examples below are listed by property used in the rule along with the corresponding operator.

 

URL using "is in list"

 

Using the property "URL", implies that you will create list entries which take into account the full URL. Using the operator "is in list" implies an exact string match.

 

2.0.0_url_isinlist.png

Good

 

Entries in "Good: URL String List"

 

Entry: http://www.mcafee.com/us/products/web-gateway.aspx

Why it's good: Full URL is used as it is needed due to "is in list" operator.

 

2.0.1_url_isinlist.png

 

Bad

 

Entries in "Bad: URL String List"

 

Entry: www.mcafee.com/us/products/web-gateway.aspx

Why it's bad: The entry doesn't include the protocol information (http://). The URL property evaluates the full URL and the operator "is in list", implies exact string match.

 

2.0.2_url_isinlist.png

 

 

URL using "matches in list"

 

Using the property "URL" implies that you will create list entries which take into account the full URL. Using the operator "matches in list" allows for wildcard matches.

 

2.1.0_url_matchesinlist.png

 

Good

 

Entries in "Good: URL Wildcard List"

 

Entry: http://www.mcafee.com/*

Why it's good: This entry contains a trailing wildcard which will allow any HTTP request to www.mcafee.com. However, it will not match on requests for http://mcafee.com/.

 

Entry: regex(htt(p|ps)://(.*\.|\.?)mcafee.com(\/.*|\/?))

Why it's good: This entry is a bit more complex as it uses regular expressions. This entry will allow any request, HTTP or HTTPS, to mcafee.com and it's subdomains.

 

Entry: regex(htt(p|ps)://(.*\.|\.?)mcafee.(com|co.uk)(\/.*|\/?))

Why it's good: This entry is the same as the previous entry but demonstrates how you can allow other top level domains, such as '.com' or '.co.uk'.

 

2.1.1_url_matchesinlist.png

 

Bad

 

Entries in "Bad: URL Wildcard List"

 

Entry: *.mcafee.com*

Why it's bad: Using this entry, the entry could match on another string within the URL, for example: hxxp://malicious-download-site.cc/malicious-file.exe?url=www.mcafee.com

     

    2.1.2_url_matchesinlist.png

     

    URL.Host using "is in list"

     

    Using the property "URL.Host" implies that you will create list entries which take into account only the domain portion of the URL. Using the operator "is in list" implies an exact string match.

     

    3.0.0_urlhost_isinlist.png

     

    Good

     

    Entry in "Good: URL.Host String List"

     

    Entry: www.mcafee.com

    Why it's good: The domain of the requested URL is 'www.mcafee.com' which is an uses exact string match.

     

    3.0.1_urlhost_isinlist.png

     

    Bad

     

    Entries in "Bad: URL.Host String List"

     

    Entry: mcafee.com

    Why it's bad: The entry value is incorrect (mcafee.com), the actual property value is 'www.mcafee.com'.

     

    Entry: *.mcafee.com

    Why it's bad: The operator is "is in list" which implies an exact string match, wildcards will not match.

     

    Entry: *.mcafee.com/us*

    Why it's bad: The URL.Host property is limited only to the domain portion of the URL, not the path (/us). In addition, the operator "is in list" which implies an exact string match, wildcards will not match.

         

        3.0.2_urlhost_isinlist.png

         

        URL.Host using "matches in list"

         

        Using the property "URL.Host" implies that you will create list entries which take into account only the domain portion of the URL. Using the operator "matches in list" allows for wildcard match.

         

        3.1.0_urlhost_matchesinlist.png

         

        Good

         

        Entries in "Good: URL.Host Wildcard List"

         

        Entry: mcafee.com

        Why it's good: This entry will not match for 'www.mcafee.com' but if you intend to allow access to mcafee.com (no www) you will need it unless you use regular expressions.

         

        Entry: *.mcafee.com

        Why it's good: This entry will match on any subdomain of mcafee.com (but not actually mcafee.com itself).

         

        Entry: regex((.*\.|\.?)mcafee.com)

        Why it's good: This single entry uses regular expressions and will allow both mcafee.com and any subdomains of mcafee.com.

         

        3.1.1_urlhost_matchesinlist.png

         

        Bad

         

        Entries in "Bad: URL.Host Wildcard List"

         

        Entry: *.mcafee.com*

        Why it's bad: Using this entry, the entry could match on another string within the URL, for example: hxxp://www.mcafee.com.malicious-download-site.cc/

         

        Entry: *.mcafee.com/us*

        Why it's bad: URL.Host property is limited only to the domain portion of the URL is acceptable, not the path (/us).

           

          3.1.2_urlhost_matchesinlist.png

           

          URL.Domain vs. URL.Host.BelongsToDomains

           

          The URL.Domain property was introduced in 7.4. It was a property designed to be more consistent with other URL related properties (URL.Host, URL, etc...). It acts nearly identically to that of URL.Host.BelongsToDomains, but does not require a list as a setting, instead the list can be the operand.

           

          URL.Domain is a string property which contains the top level domain of the requested URL (i.e. "mcafee.com").

          7.0.0_urldomain_isintlist.png

           

          URL.Host.BelongsToDomains<ListName> is a boolean property which returns true if the URL's top level domain is in the list specified on the rule (ListName). If the domain of the URL is not in the list, the property returns false.

          7.1.0_urlhost_belongs.png

           

          URL.Domain using "is in list"

           

          Using the property "URL.Domain" implies that you will create list entries which take into account just the top level domain of the URL. Using the operator "is in list" implies an exact string match.

           

          6.0.0a_urldomain_isintlist.png

           

          Good

           

          Entries in "Good: URL.Domain String List"

           

          Entry: mcafee.com

          Why it's good: URL.Domain will simply equal "mcafee.com".

           

          6.0.1_urldomain_isintlist.png

           

           

          Bad

           

          Entries in "Bad: URL.Domain String List"

           

          Entry: www.mcafee.com

          Why it's bad: URL.Domain is "mcafee.com", not "www.mcafee.com". Use URL.Host instead.

           

          Entry: *.mcafee.com

          Why it's bad: URL.Domain equals "mcafee.com", so "*." would prevent matching. "is in list" implies a string, not a wildcard.

           

          6.0.2_urldomain_isintlist.png

           

           

          URL.Domain using "matches in list"

           

          Using the property "URL.Domain" implies that you will create list entries which take into account just the top level domain of the URL. Using the operator "matches in list" allows for wildcard matches.

           

          6.1.0a_urldomain_matchesintlist.png

           

          Good

           

          Entries in "Good: URL.Domain Wildcard List"

           

          Entry: regex(mcafee.(com|co.uk))

          Why it's good: URL.Domain equals "mcafee.com" so it will match. "mcafee.co.uk" will also match.

           

          6.1.1_urldomain_matchesintlist.png

           

           

          Bad

           

          Entries in "Bad: URL.Domain Wildcard List"

           

          Entry: *.mcafee.com

          Why it's bad: URL.Domain of "mcafee.com" will not match due to the "*.".

           

          Entry: *mcafee.com

          Why it's bad: It will match on "mcafee.com", BUT it could match on "maliciousdomainmcafee.com" too.

           

          6.1.2_urldomain_matchesintlist.png

           

           

          URL.Host.BelongsToDomains

           

          The URL.Host.BelongsToDomains property was introduced in 7.2. It was designed to simplify the complexity of adding list entries. Using the property "URL.Host.BelongsToDomains" allows you to simply enter the domain of interest.

           

          So if you wish to white list all mcafee.com sites (including subdomains), you can simply enter mcafee.com, there is no need to worry about wildcards.

           

          4.0.0_urlhost_belongs.png

          Good

           

          Entries in "Good: Only Domain List"

           

          Entry: mcafee.com

          Why it's good: Using this entry, it would correctly match for all mcafee.com subdomains, including mcafee.com, www.mcafee.com, secure.mcafee.com, etc...

           

          Entry: www.mcafee.com

          Why it's good: Using this entry, it would correctly match only for www.mcafee.com subdomains. It would not allow other subdomains of the top domain 'mcafee.com'. This is useful in case you wanted to allow a subdomain, but not the entire domain.

           

          4.0.1_urlhost_belongs.png

           

          Bad

           

          Entries in "Bad: Only Domain List"

           

          Entry: *.mcafee.com

          Why it's bad: Using URL.Host.BelongsToDomains does not need wildcards, the property requires an exact domain match such as 'www.mcafee.com' or the top domain 'mcafee.com'.

             

            4.0.2_urlhost_belongs.png

             

            Test Ruleset

            You can use the test ruleset in your own environment to see how it works! Also check out the updated ruleset for version 7.4+, click here.

             

            Conclusion

            From the examples, it should be clear that the cleanest/easiest way to create domain based whitelist entries is through the use of the "URL.Host.BelongsToDomains" property. I hope this helps clarify use cases for the various URL related properties, perhaps it will help with understanding other properties as well.

            Comments (5)