cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Customer Maintained Subscribed Lists and External Lists

 

Web Gateway: Using Customer Maintained Subscribed Lists and External Lists

Introduction

McAfee Web Gateway has the capability to use lists that are maintained off-box from the Web Gateway for use in your policy. A customer can utilize a subscribed list or an external list.  One of the benefits of using a subscribed or external list is that the contents of the list can be managed on a web server or an external source by different groups within a company. Web Gateway can then collect and use this list for use in the policy/rule engine. 

Common problems that we see in the field relate to the format of the files used for these lists. Failure to use the appropriate format can cause various problems such as rule engine errors and list update issues. The purpose of this article is to show how to setup these lists in the correct format and how to troubleshoot potential problems.

 

Subscribed Lists versus External List

 

 

Subscribed List External List

Key Characteristics

- List is declared in MWG

- List content is stored in MWG Storage

- Behaves like any internal list, only content is not edited but periodically reloaded from external sources.

 

- List is not declared and only exists externally

- Dynamic reference to the lists from the policy engine

- Cache of used lists with a defined TTL or size limit and LRU.

 

Advantages

- List content is maintained externally

- Admin can see the list content in the UI

- List is immediately available after restart/crash

- No delay when accessing the list, update happens asynchronously.

 

- Best scalability to any number of lists in a cluster.

(Example: In an ISP/SaaS environment tens of thousands of lists can exist externally and a node will only load those lists that are required for the current transactions on this node)

 

Disadvantages

- Does not scale well to thousands of lists

 

- Without a self-designed prefetch strategy, additional latency when accessing lists that are not in cache.

- Cache is lost if MWG restarts/crashes

- Lists cannot be viewed in the UI (could be shown via error message template in the browser thru special internal request though)

 

 

 

Subscribed Lists

Subscribed lists are more static in nature and are updated on a configurable schedule. Changes to the list are maintained on the web server or location where the list is stored, not on the Web Gateway.

Note: Only the list configuration is saved in a Web Gateway backup file and not the actual contents of the list.

A common use for subscribed lists is to aide in helpdesk tasks. For example, helpdesk engineers can maintain whitelist or blacklists off-box instead of locally on the Web Gateway. This could be useful if you do not want to give any Web Gateway UI rights to a specific user.

 

Setup Instructions

 

1) STEP 1: Format the list properly for the Web Gateway.

Subscribed lists can be simple (txt and xml) or complex (xml). We'll show you some examples of each below.

  • Simple Lists

          a) Text

          Here is the format of a Simple Text File:

          type=list type

          "data item 1" "data item 1 comment"

          "data item 2" "data item 2 comment"

          When creating your simple list, you'll need to substitute in the appropriate list type. The bold text below indicates the list type you'll need to specify:

      •           Application Name = applcontrol
      •           Dimension = dimension
      •           String = string
      •           Category = category
      •           IP = ip
      •           IPRange = iprange
      •           MediaType = mediatype
      •           Number = number
      •           Wildcard Expression = regex

          EXAMPLES:

          type=category

                  com.scur.category.115

                  com.scur.category.166

          NOTE: Trusted Source category codes can be downloaded here: McAfee TrustedSource Web Database Reference Guide.

 

          type=applcontrol

          "Google News"

          "Ebay.com" "Ray asked me to add this application"

          NOTE: For a listing of  Applications, you can view the System List in the Web Gateway UI: (Policy > Lists > Systems Lists > Application Name)

 

          type=regex

          "*.espn.com/nfl*" "HD-0002 - VP wants to check NFL scores"

          "*.ebay.com/auctions*"

 

          type=string

          "www.google.com" "HD-0003 - Jon needs access"

          "www.startribune.com" "HD-0004 - Tim needs to view"

 

          type=string

          "Hello\"Michael\" \"Michael!\"" ""

          "*empty comment\"\"\" ""

          NOTE: The escape character \ is used to allow a double quote as part of the data or comments as seen above.

 

          b) XML

          Here is the format of a Simple XML File:

          <content type="list type">

             <listentry>

                <entry> data item 1 </entry>

                <description> data item 1 comment</description>

             </listentry>

             <listentry>

                <entry> data item 2 </entry>

                <description> data item 2 comment</description>

             </listentry>

          </content>

 

          <content type="regex">

              <listEntry>

                 <entry>*.windowsupdate.com</entry>

                 <description>HD Ticket: 1234</description>

              </listEntry>

              <listEntry>

                 <entry>*.microsoft.com</entry>

                 <description>HD Ticket: 1235</description>

              </listEntry>

           </content>

 

  • Complex Lists

           Complex lists can only use the .xml file format. The first line of any complex list will begin with a line that looks like this:

           <content type="complex.nexthopproxy">

            The content type, in this example nexthopproxy, relates to the type of complex list you're creating. Below is a list of content types you can use.

            Note: The bold text below indicates the content type syntax you will need to use.

      •            Certificate Authority = complex.ca
      •            Extended List = complex.extendedlist
      •            Element = complex.element
      •            HostAndCertificate = complex.hostandcertificate
      •            ICAP Server = complex.icapserver
      •            NextHopProxy = complex.nexthopproxy
      •            MapType = complex.maptype

           Due to the complexity of the format, it is recommended that you use the following steps to create your complex list:

               
  1. Create a sample complex List: Use the Web Gateway to create a complex list of the 'type' you want. In the Web Gateway UI, go to: PolicyLists > Add > give the list a name and choose a content type. Press the OK button. Lastly, add content to the list and Save changes.
  2.            
  3. Export the complex List: In the Web Gateway UI, go to: Policy > Lists- Select the list and then click the Export button. Save the list to your local machine.
  4.            
  5. Modify the list: The exported list adds extra lines that we will need to remove. Edit the exported list in a text editor.
  6.            
  7. Delete all lines in the file that precede the opening <content>tag and follow the closing </content>tag.
  8.            
  9. Modify the opening <content>tag to read <content type="complex.content type">, for example <content type="complex.nexthopproxy">

 

           Here is what an exported and modified next hop proxy complex list example would look like:

           <content type="complex.nexthopproxy">

             <listEntry>

             <complexEntry defaultRights="2">

              <configurationProperties>

                <configurationProperty key="name" type="com.scur.type.string" value="Default Next Hop Proxy" />

                <configurationProperty key="host" type="com.scur.type.string" value="10.10.79.50" />

                <configurationProperty key="port" type="com.scur.type.string" value="9090" />

                <configurationProperty key="user" type="com.scur.type.string" value="" />

                <configurationProperty key="password" type="com.scur.type.string" value="" />

                <configurationProperty key="retries" type="com.scur.type.string" value="1" />

                <configurationProperty key="waittime" type="com.scur.type.string" value="10" />

                <configurationProperty key="persistent" type="com.scur.type.boolean" value="true" />

               </configurationProperties>

              </complexEntry>

              <description />

             </listEntry>

           </content>

 

           Here is a map type complex subscribed list example:

           <content type="complex.maptype">

             <listEntry>

               <complexEntry defaultRights="2">

                 <configurationProperties>

                   <configurationProperty key="key" type="com.scur.type.string" value="bob" />

                   <configurationProperty key="value" type="com.scur.type.string" value="pas$w0rD" />

                 </configurationProperties>

               </complexEntry>

               <description />

             </listEntry>

           </content>

 

STEP 2: Add list configuration to the Web Gateway

Two options for adding a list:

           a) option 1:

list1.JPG

 

 

 

 

 

 

 

 

 

 

 

 

 

           - Navigate to Policy> Lists> Add List (green circle icon)

           - Enter a name for the list and select list type. Note: the type selected here must match the actual list you created. Otherwise you will get an error when trying to save changes.

           - Select "List Content is managed remotely"

           - Select "Customer Maintained List"

           - Click "Setup"

            - Setup window opens:

list2.JPG

 

           Setup options explained:

           - URL to download list from: HTTP, HTTPS, or FTP supported

           - Use this "Certificate Authority Chain": select if HTTPS used and import certificate in .PEM format from local workstation

           - Ignore certificate errors: select for HTTP or FTP, or if you wish to ignore certificate errors on HTTPS

           - URL Authentication: can leave blank if none required to access URL for download

           - Proxy: Only configure if there is an upstream proxy

           - List content update: Select how often you want the Web Gateway to update the list

            - When done configuring your list, click "OK".

           - If successful, you will see your list appended with the label "Customer maintained" and list contents will be displayed.

list3.JPG

 

          b) option 2:

           - Navigate to Policy> Rule Sets

           - Edit an existing rule or create a new one

           - For "Rule Criteria", edit or add criteria that matches your list type, for example, URL.Categories.

           - Click "Add List..."

           - Then fill out customer maintained list according to the setup procedures shown in option 1.

 

Example of using a customer maintained subscribed list in a rule.

example-rule-cust-maintained1.JPG 

 

Additional settings

  • Navigate to Configuration> Appliances> Appliances> yourAppliance> Central Management> Advanced Subscribed Lists Settings.
  • Ensure that “Allow to download customer subscribed lists” is selected.
  • Set the following values:
    • Maximum number of allowed threads
    • Maximum number of allowed entries in list
    • Maximum list size

 

Troubleshooting Subscribed Lists 

The following logs and methods can be used to discover problems with your customer maintained subscribed list.

1) Logs showing errors, successful updates, and changes to list configuration:

           - Troubleshooting> Log files> update> customer_maintained_lists.log

2) Logs showing if elements in the list are rejected:

           - Dashboard > Alerts. You may see an error such as "Could not parse 1 list element(s) for list YOUR-LIST-NAME of type YOUR-LIST-TYPE. Invalid elements are ignored."

           - Troubleshooting> Log files> mwg-errors> mwg-core.errors.log

3) Logs showing which user created or deleted a customer maintained subscribed list (search for the list name to find quickly):

           - Troubleshooting> Log files> audit> audit.log

4) Setup Incident Alert(s) for the following IDs. See Best Practice guide "Web Gateway: Configuring Incident Notifications and Alerts".

           - 250 (An entry in a list is invalid and will be ignored).

           - 323 (Update of customer subscribed list x succeeded on node y.)

           - 324 (Update of customer subscribed list x failed on nodes y, z, ...)

           - 324 (Status of customer subscribed list x on node y is up to date.)

           - 326 (Download of customer subscribed list x failed on nodes y, z, ...)

 

External Lists

An external list retrieves data from a data source like a web server, file on disk, LDAP, or database and then converts them into the appropriate data type by the use of properties in the Web Gateway. An important benefit of external lists is that they are processed and fetched dynamically on the appliance. All retrieving and conversion of external list data is triggered when the data is first used in a rule in your policy.

Setup Instructions:

  • Navigate to Policy> Settings> Engines> External Lists> right-click> Add.
  • Enter name for list.
  • Select data source type. Each type will have its own required fields. Follow instructions on the screen.

external-list1.jpg

 

Data Source Types:

 

        1) Web Service

           A web service will be accessed under HTTP, HTTPS, or FTP protocol. The file format will need to be in XML, JSON, or Text format.

               a) XML

               The format of this file simply needs to follow typical XML standards. Web Gateway can use a standard called 'XPath' that will allow it to navigate through elements of the XML file it is retrieving and filter for specific data you want populated in the external list.  Additional information about creating an Xpath can be found at http://www.w3schools.com/xpath/.

               XML file example:

               <user-info>

                 <user>

                   <name lang="eng">tom</name>

                   <level>1</level>

                 </user>

                 <user>

                   <name lang="eng">bob</name>

                   <level>5</level>

                 </user>

               </user-info>

               Using the the xml example above, we will show you how to use xPath to retrieve certain elements of the XML file.

            - Settings under Policy> Settings> Engines> External Lists:

                 XPath Expression  =  /user-info/user/name/text()

                This XPath expression will give the name of the user in the XML file. Essentially, this expression will search down the XML file looking for each tag you've specified in the path above and then use the 'text()' function to only pull out the text value of the <name> tag.

                 XPath Expression  =  /user-info/user/level/text()

                 This XPath expression will filter out the text value of the <level> tag for use in the external list.

 

           b) JSON

           The data source will return data as JSON object.

            - JSON examples can be found starting on page 152 in the product guide.

           - Use of parameters (placeholders) in the rules is explained starting on page 139 in the product guide.

           - Example using external JSON file list in rules:

example-rule-external-list4.JPG 

example-rule-external-list3.JPG

            c) TXT

           The format of this file need to simply be a text file. For text files a regular expression is used to locate the data for our list.

           Text file example:

           Member Group name

           ------ ----------

           group1 admin

           group1 sudoadmin

           group2 report

           group3 users

           group4 guest

           Added sudoadmin group. Updated on 9/28/2017

           - The regular expression below uses a grouping operator so only the "group name" column will be retrieved for our list.

           - Settings under Policy> Settings> Engines> External Lists:

               Data Type                    =  Plain Text

               Regular Expression    =  regex(^group. (.*))

 

           - Example using external text file list in rules:

example-rule-external-list2.JPG

 

example-rule-external list1.JPG

 

           2) File on disk

           File on disk refers to a file within your local file system (on the Web Gateway appliance itself). This option is good for testing your external list when another external source is not yet available. Accepted list formats are the same as for web service data sources (XML, JSON, and TEXT).

           

           3) LDAP or LDAPS

           Another data source for external lists is LDAP. Once connected to a LDAP server, you can specify a LDAP Search Filter and Attribute to fetch specific data for the list.  The search filter is a LDAP standard and the attribute will dependent on your LDAP configuration. Below is an example that searches for the user bob and returns the 'Title' attribute for that user.

           - Settings under Policy> Settings> Engines> External Lists:

                Data Type        =  LDAP

                Search Filter    = (sAMAccountName=bob)

                Attribute          = Title

 

           4) Database

           Another data source for external lists is Database. You can choose between retrieving data from a PostgreSQL or SQLite3. Once connected to the database, you can specify a SQL query to fetch specific data for the list. Below is an example SQL query used with External Lists.

           - Settings under Policy> Settings> Engines> External Lists:

                Data Type        =  Database

                SQL Query      =  SELECT user from userTable;

 

Troubleshooting External Lists 

The following logs and methods can be used to discover problems with your external list.

1) Dashboard errors.

           - Dashboard > Alerts.

           - Look for errors that include “External Lists Filter”, for example:

                       "No resource for data source is available (Origin: External Lists Filter, ID: 1806)"

2) Error logs.

           - Troubleshooting> Log files> mwg-errors> mwg-core.errors.log

                      a) Search for errors that include “ExternalListsFilter”, for example:

                           [ExternalListsFilterPlugin] [ExtListsError] Error in ExternalListsFilter: 'No such resource (file, URL, LDAP server, etc.)'.

                           [ExternalListsFilterPlugin] [ExtListsError] Error in ExternalListsFilter: 'CFileDataSource: file don't exists! '/opt/mwg''.

           - If extra error logs was created, for example, such as the one created with the YouTube External list, it would be located here: Troubleshooting> Log files> user-defined-logs.

3) Logs showing which user created or deleted an external list (search for ‘External Lists” to find quickly):

           - Troubleshooting> Log files> audit> audit.log

4) Check to make sure the Web Gateway can access web service data source. For example, when checking if the Web Gateway can retrieve the YouTube external list, run the following command on the CLI. If successful, you will see the output “Connected to…”.

           curl -v https://www.googleapis.com

5) Setup Incident Alert(s) for the following IDs. See Best Practice guide "Web Gateway: Configuring Incident Notifications and Alerts".

           - 1800 (The number of entries that can be retrieved from an external list has exceeded the configured limit.).

           - 1801 (The amount of data of entries that can be retrieved from an external list has exceeded the configured limit.)

           - 1802 (An error occurred when data was retrieved from an external list.)

           - 1803 (An error occurred when data that had been retrieved from an external list was converted.)

           - 1804 (A time-out occurred when data was retrieved from an external list.)

           - 1805 (Permission to retrieve data from an external list was denied.)

           - 1806 (A resource for retrieving external list data could not be found.)

 

 

Labels (1)
Attachments
Comments

Just several comments:

  • in 7.4.0 - for Web & File data source, the new format was added - JSON. In this case, you can read complex data structure into JSON data type, and work with it. It's very useful for different databases, like CouchDB, MongoDB, etc.
  • Since 7.3.2 there is a support for Map Type that consists from pair of key/value. To do this, data source need to specify additional information:
  1. for databases - query should return 2 columns
  2. for LDAP - you need to specify second attribute for search
  3. for XML - you specify 2nd XPath expression that should return the same number of entries as 1st one
  4. for Text format you need to specify regex with 2 capturing groups - for example, if you have a file with entries like: "user=password", then you can split it into Map using regex like: "^([^=]*)=(.*)$"

To be clear, adding and removing entries from either list type will take affect the next time the proxy queries the externally hosted list, correct?

100% correct on the updates. updates to the lists on the external servers will come in next time mwg checks for updates

How to use ExtLists.CategoryList?

I tried to use it with Data Source Type "Web Service". The Webserver responds with following:

General News

Messaging

Problem: There is no content within the Property. If I use use WildcardsList or StringList instead, there is no problem. Sees Categories are handled in a special way. But I there is no description about how to handle. Please help.

Just found the answer by myself. You have to give the category id not the category string.

Category ids see http://www.trustedsource.org/download/ts_wd_reference_guide.pdf

Are there any limitations on the list? How many lines can be in the simple list file?

There seems to be this limitation:


There is a restriction in size for subscribed lists. A subscribed list must not be larger than 4 MB or


contain more than 100,000 entries.


Does anyone know how to increase this or if there is a workaround?

The limits can be set:

Configuration >> Appliances >> <hostname> >> Central Management >> Advanced Subscribed Lists Settings

Thanks that worked!

However after the list is imported we get a Java error "Java heap space". I believe the list is too big to be displayed in the management console? Has anyone faced that error?

Java ist limited in memory consumption. Google for How to increase Java Heap Space. eg How to Increase Java Memory in Windows 7: 9 Steps

Hm.. this is bit confusing for me.

I have some daily list of suspicious IP address like below:

 

46.109.168.179
107.180.56.147
114.44.192.128
118.170.130.207
188.118.2.26
31.184.193.179
38.229.70.4
74.208.192.75
81.183.56.217
91.198.174.192
91.223.88.205
93.184.220.29

How do I enter into that list daily or weekly so that it is blocked by MWG proxy?

Contributors
Version history
Revision #:
5 of 5
Last update:
‎05-01-2019 06:48 AM
Updated by:
 

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community