cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Subscribed Lists and External Lists Format Examples

Introduction

McAfee Web Gateway has the capability to use lists that are maintained off-box from the Web Gateway for use in your policy. A customer can utilize a subscribed list or an external list.  One of the benefits of using a subscribed or external list is that the contents of the list can be managed on a web server or an external source by different groups within a company. Web Gateway can then collect and use this list for use in the policy/rule engine.

 

Common problems that we see in the field relate to the format of the files used for these lists. Failure to use the appropriate format can cause various problems such as rule engine errors and list update issues. The purpose of this article is to focus on the format of Subscribed and External lists you must use in order for the Web Gateway to understand and utilize these lists.We'll describe the formats in detail and also provide some samples that are attached to this article.

 

 

Subscribed Lists

This section will focus on customer maintained subscribed lists. If you're curious about McAfee Maintained subscribed lists, you can read more here:

 

Subscribed lists are more static in nature and are updated on a configurable schedule. Changes to the list are maintained on the web server or location where the list is stored.

Note: only the list configuration is saved in a Web Gateway backup file and not the actual contents of the list.

 

A common use for subscribed lists is to aide in helpdesk tasks. For example, helpdesk engineers can maintain whitelist or blacklists off-box instead of locally on the Web Gateway. This could be useful  if you do not want to give any Web Gateway UI rights to a specific user.

 

Subscribed lists can be simple (txt and xml) or complex (xml). We'll show you some examples of each below.

 

Simple Lists

 

Text

Here is the format of a Simple Text File:

type=list type

"data item 1" "data item 1 comment"

"data item 2" "data item 2 comment"

 

When creating your simple list, you'll need to substitute in the appropriate list type. The bolded text below indicates the list type you'll need to specify:

      • Application Name = applcontrol
      • Dimension = dimension
      • String = string
      • Category = category
      • IP = ip
      • IPRange = iprange
      • MediaType = mediatype
      • Number = number
      • Wildcard Expression = regex

        type=applcontrol

        "Google News"

        "Ebay.com" "Ray asked me to add this application"

         

        NOTE: For a listing of  Applications, you can view the System List in the Web Gateway UI: (Policy > Lists > Systems Lists > Application Name)

         

        type=regex

        "*.espn.com/nfl*" "HD-0002 - VP wants to check NFL scores"

        "*.ebay.com/auctions*"

         

        type=string

        "www.google.com" "HD-0003 - Jon needs access"

        "www.startribune.com" "HD-0004 - Tim needs to view"

         

        type=string

        "Hello\"Michael\" \"Michael!\"" ""

        "*empty comment\"\"\" ""

         

        NOTE: The escape character \ is used to allow a double quote as part of the data or comments as seen above.

 

XML

Here is the format of a Simple XML File:

<content type="list type">

   <listentry>

      <entry> data item 1 </entry>

      <description> data item 1 comment</description>

   </listentry>

   <listentry>

      <entry> data item 2 </entry>

      <description> data item 2 comment</description>

   </listentry>

</content>

 

<content type="regex">

   <listEntry>

      <entry>*.windowsupdate.com</entry>

      <description>HD Ticket: 1234</description>

   </listEntry>

   <listEntry>

      <entry>*.microsoft.com</entry>

      <description>HD Ticket: 1235</description>

   </listEntry>

</content>

 

 

 

Complex Lists

Complex lists can only use the .xml file format. The first line of any complex list will begin with a line that looks like this:

<content type="complex.nexthopproxy">

 

The content type, in this example nexthopproxy, relates to the type of complex list you're creating. Here are a list of content types you can use:

 

Note: The bolded text below indicates the content type syntax you will need to use:

    • Certificate Authority = complex.ca
    • Extended List = complex.extendedlist
    • Element = complex.element
    • HostAndCertificate = complex.hostandcertificate
    • ICAP Server = complex.icapserver
    • NextHopProxy = complex.nexthopproxy
    • MapType = complex.maptype

 

Due to the complexity of the format,  it is recommended that you use the following steps to create your complex list:

 

1. Create a sample complex List: Use the Web Gateway to create a complex list of the 'type' you want. In the Web Gateway UI, go to: Policy > Lists > Add > give the list a name and choose a content type. Press the OK button. Lastly, add content to the list and Save changes.

2. Export the complex List: In the Web Gateway UI, go to: Policy > Lists - Select the list and then click the Export button. Save the list to your local machine.

3. Modify the list: The exported list adds extra lines that we will need to remove. Edit the exported list in a text editor.

a. Delete all lines in the file that precede the opening <content> tag and follow the closing </content> tag.

b. Modify the opening <content> tag to read <content type="complex.content type">, for example <content type="complex.nexthopproxy">

 

Here is what an exported and modified next hop proxy complex list example would look like:

<content type="complex.nexthopproxy">

  <listEntry>

  <complexEntry defaultRights="2">

   <configurationProperties>

     <configurationProperty key="name" type="com.scur.type.string" value="Default Next Hop Proxy" />

     <configurationProperty key="host" type="com.scur.type.string" value="10.10.79.50" />

     <configurationProperty key="port" type="com.scur.type.string" value="9090" />

     <configurationProperty key="user" type="com.scur.type.string" value="" />

     <configurationProperty key="password" type="com.scur.type.string" value="" />

     <configurationProperty key="retries" type="com.scur.type.string" value="1" />

     <configurationProperty key="waittime" type="com.scur.type.string" value="10" />

     <configurationProperty key="persistent" type="com.scur.type.boolean" value="true" />

    </configurationProperties>

   </complexEntry>

   <description />

  </listEntry>

</content>

 

Here is a map type complex subscribed list example:

<content type="complex.maptype">

  <listEntry>

    <complexEntry defaultRights="2">

      <configurationProperties>

        <configurationProperty key="key" type="com.scur.type.string" value="bob" />

        <configurationProperty key="value" type="com.scur.type.string" value="pas$w0rD" />

      </configurationProperties>

    </complexEntry>

    <description />

  </listEntry>

</content>

 

 

External Lists

An external list retrieves data from a data source like a web server, file on disk, LDAP, or database and then converts them into the appropriate data type by the use of properties in the Web Gateway. An important benefit of external lists is that they are processed and fetched dynamically on the appliance. All retrieving and conversion of external list data is triggered when the data is first used in a rule in your policy.

 

A common use for external lists are rules related to restricting/allowing access to youtube content. A good example can be found here. Note: credentials for this portal are needed.

 

Web Service | File on Disk

When using a Web service or File on Disk data source type, the file format will need to be in XML or Text format.

 

XML

The format of this file simply needs to follow typical XML standards. The interesting part to note is that Web Gateway is able to use a standard called 'XPath' that will allow the Web Gateway to navigate through elements of the XML file it is retrieving and filter for specific data you want populated in the external list.  Additional information about creating an Xpath can be found at http://www.w3schools.com/xpath/.

 

XML file example:

<user-info>

  <user>

    <name lang="eng">tom</name>

    <level>1</level>

  </user>

  <user>

    <name lang="eng">bob</name>

    <level>5</level>

  </user>

</user-info>

 

Using the the xml example above, we will show you how to use xPath to retrieve certain elements of the XML file.

 

Web Gateway External list Settings

XPath Expression  =  /user-info/user/name/text()

This XPath expression will give the name of the user in the XML file. Essentially, this expression will search down the XML file looking for each tag you've specified in the path above and then use the 'text()' function to only pull out the text value of the <name> tag.

 

XPath Expression  =  /user-info/user/level/text()

This XPath expression will filter out the text value of the <level> tag for use in the external list.

 

 

Text

The format of this file need to simply be a text file. For text files a regular expression is used to locate the data for our list.

 

Text file example:

Member Group name

------ ----------

group1 admin

group1 sudoadmin

group2 report

group3 users

group4 guest

Added sudoadmin group. Updated on 9/28/2017

The regular expression below uses a grouping operator so only the "group name" column will be retrieved for our list.

 

Web Gateway External list Settings:

    Data Type                    =  Plain Text

    Regular Expression    =  regex(^group. (.*))

 

 

LDAP

Another data source for external lists is LDAP. Once connected to a LDAP server, you can specify a LDAP Search Filter and Attribute to fetch specific data for the list.  The search filter is a LDAP standard and the attribute will dependent on your LDAP configuration. Below is an example that searches for the user bob and returns the 'Title' attribute for that user..

 

Web Gateway External list Settings:

Data Type        =  LDAP

Search Filter    = (sAMAccountName=bob)

Attribute          = Title

 

 

Database

Another data source for external lists is Database. You can choose between retrieving data from a PostgreSQL or SQLite3. Once connected to the database, you can specify a SQL query to fetch specific data for the list. Below is an example SQL query used with External Lists.

 

Web Gateway External list Settings:

Data Type        =  Database

SQL Query      =  SELECT user from userTable;

Labels (1)
Attachments
Comments

Just several comments:

  • in 7.4.0 - for Web & File data source, the new format was added - JSON. In this case, you can read complex data structure into JSON data type, and work with it. It's very useful for different databases, like CouchDB, MongoDB, etc.
  • Since 7.3.2 there is a support for Map Type that consists from pair of key/value. To do this, data source need to specify additional information:
  1. for databases - query should return 2 columns
  2. for LDAP - you need to specify second attribute for search
  3. for XML - you specify 2nd XPath expression that should return the same number of entries as 1st one
  4. for Text format you need to specify regex with 2 capturing groups - for example, if you have a file with entries like: "user=password", then you can split it into Map using regex like: "^([^=]*)=(.*)$"

To be clear, adding and removing entries from either list type will take affect the next time the proxy queries the externally hosted list, correct?

100% correct on the updates. updates to the lists on the external servers will come in next time mwg checks for updates

How to use ExtLists.CategoryList?

I tried to use it with Data Source Type "Web Service". The Webserver responds with following:

General News

Messaging

Problem: There is no content within the Property. If I use use WildcardsList or StringList instead, there is no problem. Sees Categories are handled in a special way. But I there is no description about how to handle. Please help.

Just found the answer by myself. You have to give the category id not the category string.

Category ids see http://www.trustedsource.org/download/ts_wd_reference_guide.pdf

Are there any limitations on the list? How many lines can be in the simple list file?

There seems to be this limitation:


There is a restriction in size for subscribed lists. A subscribed list must not be larger than 4 MB or


contain more than 100,000 entries.


Does anyone know how to increase this or if there is a workaround?

The limits can be set:

Configuration >> Appliances >> <hostname> >> Central Management >> Advanced Subscribed Lists Settings

Thanks that worked!

However after the list is imported we get a Java error "Java heap space". I believe the list is too big to be displayed in the management console? Has anyone faced that error?

Java ist limited in memory consumption. Google for How to increase Java Heap Space. eg How to Increase Java Memory in Windows 7: 9 Steps

Hm.. this is bit confusing for me.

I have some daily list of suspicious IP address like below:

 

46.109.168.179
107.180.56.147
114.44.192.128
118.170.130.207
188.118.2.26
31.184.193.179
38.229.70.4
74.208.192.75
81.183.56.217
91.198.174.192
91.223.88.205
93.184.220.29

How do I enter into that list daily or weekly so that it is blocked by MWG proxy?

Contributors
Version history
Revision #:
2 of 2
Last update:
‎03-20-2018 01:12 PM
Updated by: