Subscribed Lists and External Lists Format Examples

Version 1


    Introduction

     

    Mcafee Web Gateway has the capability to use lists that are maintained off-box from the Web Gateway for use in your policy. A customer can utilize a subscribed list or an external list.  One of the benefits of using a subscribed or external list is that the contents of the list can be managed on a web server or an external source by different groups within a company. Web Gateway can then collect and use this list for use in the policy/rule engine.

     

    Common problems that we see in the field relate to the format of the files used for these lists. Failure to use the appropriate format can cause various problems such as rule engine errors and list update issues. The purpose of this article is to focus on the format of Subscribed and External lists you must use in order for the Web Gateway to understand and utilize these lists.We'll describe the formats in detail and also provide some samples that are attached to this article.


    Subscribed Lists

     

    This section will focus on customer maintained subscribed lists. If you're curious about Mcafee Maintained subscribed lists, you can read more here: https://community.mcafee.com/docs/DOC-4771 .

     

    Subscribed lists are more static in nature and are updated on a configurable schedule. Changes to the list are maintained on the web server or location where the list is stored.

    Note: only the list configuration is saved in a Web Gateway backup file and not the actual contents of the list.

     

    A common use for subscribed lists is to aide in helpdesk tasks. For example, helpdesk engineers can maintain whitelist or blacklists off-box instead of locally on the Web Gateway. This could be useful  if you do not want to give any Web Gateway UI rights to a specific user.

     

    Subscribed lists can be simple (txt and xml) or complex (xml). We'll show you some examples of each below.

     

     

     

     

    Simple Lists

     

    Text


    Here is the format of a Simple Text File:

     

    type=list type

    “data item 1” “data item 1 comment”

    “data item 2” “data item 2 comment”

     

    When creating your simple list, you'll need to substitute in the appropriate list type. The bolded text below indicates the list type you'll need to specify:


     

             Application Name = applcontrol

             Dimension = dimension

             String = string

             Category = category

             IP = ip

             IPRange = iprange

             MediaType = mediatype

             Number = number

             Wildcard Expression = regex


     

    type=applcontrol
    "Google News"
    "Ebay" "Ray asked me to add this application"
    
    NOTE: For a listing of  Applications, you can view the System List in the Web Gateway UI: (Policy > Lists > Systems Lists > Application Name)
    
    type=regex
    "*.espn.com/nfl*" "HD-0002 - VP wants to check NFL scores"
    "*.ebay.com/auctions*"
    
    
    type=string
    "www.google.com" "HD-0003 - Jon needs access"
    "www.startribune.com" "HD-0004 - Tim needs to view" 
    
    type=string
    "Hello\"Michael\" \"Michael!\"" ""
    "*empty comment\"\"\" ""
    
    NOTE: The escape character \ is used to allow a double quote as part of the data or comments as seen above.
    
    

     

    XML

     

    Here is the format of a Simple XML File:

     

     <Content type=”list type”>
      <listentry>
        <entry> data item 1 </entry>
        <description> data item 1 comment</description> 
      </listentry>
      <listentry>
        <entry> data item 2 </entry>
        <description> data item 2 comment</description> 
      </listentry>
    </content>
         
    <content type="regex">
      <listEntry>
        <entry>*.windowsupdate.com</entry>
        <description>HD Ticket: 1234</description>
      </listEntry>
      <listEntry>
        <entry>*.microsoft.com</entry>
        <description>HD Ticket: 1235</description>
      </listEntry>
    </content>
    

     

     

     

     

           

     

    Complex Lists

     

    Complex lists can only use the .xml file format. The first line of any complex list will begin with a line that looks like this:

     

    <content type="complex.nexthopproxy">

     

    The content type, in this example nexthopproxy, relates to the type of complex list you're creating. Here are a list of content types you can use:

     

    Note: The bolded text below indicates the content type syntax you will need to use:

     

    Certificate Authority = complex.ca

    Extended List = complex.extendedlist

    Element = complex.element

    HostAndCertificate = complex.hostandcertificate

    ICAP Server = complex.icapserver

    NextHopProxy = complex.nexthopproxy

    MapType = complex.maptype

     

     

    Due to the complexity of the format,  it is recommended that you use the following steps to create your complex list:

     

          1. Create a sample complex List: Use the Web Gateway to create a complex list of the 'type' you want. In the Web Gateway UI, go to: Policy > Lists > Add > give the list a name and choose a content type. Press the OK button. Lastly, add content to the list and Save changes.
          2. Export the complex List: In the Web Gateway UI, go to: Policy > Lists - Select the list and then click the Export button. Save the list to your local machine.
          3. Modify the list: The exported list adds extra lines that we will need to remove. Edit the exported list in a text editor.
                  A. Delete all lines in the file that precede the opening <content> tag and follow the closing </content> tag.

                  B. Modify the opening <content> tag to read <content type="complex.content type">, for example <content type="complex.nexthopproxy">

     

    Here is what an exported and modified next hop proxy complex list example would look like:


     

     

    <content type="complex.nexthopproxy">
    <listEntry>
      <complexEntry defaultRights="2">
       <configurationProperties> 
         <configurationProperty key="name" type="com.scur.type.string" value="Default Next Hop Proxy" />
         <configurationProperty key="host" type="com.scur.type.string" value="10.10.79.50" />
         <configurationProperty key="port" type="com.scur.type.string" value="9090" />  
         <configurationProperty key="user" type="com.scur.type.string" value="" />
         <configurationProperty key="password" type="com.scur.type.string" value="" />
         <configurationProperty key="retries" type="com.scur.type.string" value="1" />
         <configurationProperty key="waittime" type="com.scur.type.string" value="10" />
         <configurationProperty key="persistent" type="com.scur.type.boolean" value="true" />
        </configurationProperties>
       </complexEntry>
       <description />
    </listEntry>
    </content>
    

     


     

     

    Here is a map type complex subscribed list example:


     

    <content type="complex.maptype">
    <listEntry>
      <complexEntry defaultRights="2">
       <configurationProperties>
         <configurationProperty key="key" type="com.scur.type.string" value="bob" />
         <configurationProperty key="value" type="com.scur.type.string" value="pas$w0rD" />
        </configurationProperties>
       </complexEntry>
       <description />    
    </listEntry>  
    </content>
    


     

    External Lists

     

     


    An external list retrieves data from a data source like a web server, file on disk,  LDAP, or database and then converts them into the appropriate data type by the use of properties in the Web Gateway. An important benefit of external lists is that they are processed and fetched dynamically on the appliance. All retrieving and conversion of external list data is triggered when the data is first used in a rule in your policy.

     

    A common use for external lists are rules related to restricting/allowing access to youtube content. A good example can be found here. Note: credentials for this portal are needed.


    Web Service | File on Disk


    When using a Web service or File on Disk data source type, the file format will need to be in XML or Text format.

     

    XML

     

     

     

     

     

     

     

    The format of this file simply needs to follow typical XML standards. The interesting part to note is that Web Gateway is able to use a standard called 'XPath' that will allow the Web Gateway to navigate through elements of the XML file it is retrieving and filter for specific data you want populated in the external list.  Additional information about creating an Xpath can be found at http://www.w3schools.com/xpath/.

     

     

    XML file example:

     

     

    <user-info>
      <user>
        <name lang="eng">tom</name>
        <level>1</level>
      </user>
    
    
      <user>
        <name lang="eng">bob</name>
        <level>5</level>
      </user>
    </user-info>
    

     

     

    Using the the xml example above, we will show you how to use xPath to retrieve certain elements of the XML file.

     


    Web Gateway External list Settings:

     

    XPath Expression  =  /user-info/user/name/text()


    This XPath expression will give the name of the user in the XML file. Essentially, this expression will search down the XML file looking for each tag you've specified in the path above and then use the 'text()' function to only pull out the text value of the <name> tag.

     

                     
    XPath Expression  =  /user-info/user/level/text()


    This XPath expression will filter out the text value of the <level> tag for use in the external list.

     

    Text

     

     

     

    The format of this file need to simply be a text file. For text files a regular expression is used to locate the data for our list.

     

     

    Text file example:

    Member Group name
    ------ ----------
    group1 admin
    group1 sudoadmin
    group2 report
    group3 users
    group4 guest
    Added sudoadmin group. Updated on 9/28/2012
    
    

     

    The regular expression below uses a grouping operator so only the "group name" column will be retrieved for our list.

     

    Web Gateway External list Settings:

     

     

    Data Type                    =  Plain Text
    Regular Expression  =  regex(^group. (.*))

     

     

     

    LDAP

     

     

    Another data source for external lists is LDAP. Once connected to a LDAP server, you can specify a LDAP Search Filter and Attribute to fetch specific data for the list.  The search filter is a LDAP standard and the attribute will dependent on your LDAP configuration. Below is an example that searches for the user bob and returns the 'Title' attribute for that user..

     

    Web Gateway External list Settings:

    Data Type        =  LDAP

    Search Filter   = (SamAccountName=bob)

    Attribute           = Title

     

     

     

    Database

    Another data source for external lists is Database. You can choose between retrieving data from a PostgreSQL or SQLite3. Once connected to the database, you can specify a SQL query to fetch specific data for the list. Below is an example SQL query used with External Lists.

     

    Web Gateway External list Settings:

     

    Data Type        =  Database

    SQL Query      =  SELECT user from userTable;