Rule Engine Tracing

Version 6

     

    What is Rule Engine Tracing?

     

     

    With MWG 7.3.2, McAfee introduced the new "Rule Tracing Central" feature in the Troubleshooting section of the Web Gateway UI. Rule Engine Tracing is a graphical map of the path a request and its response take through your policy.

     

    Rule Tracing Central was designed to make debugging your rules easy, help you understand the flow a web request takes through the rule engine, and save you time!

     

    Have you ever needed to find out why a site was getting blocked or why your whitelist isn't working correctly? Did you ever have to figure out why a user was not assigned to the correct rules in your Policy? Rule Engine Tracing will help you solve all of these problems.

     

     

    In this document you will gain a basic understanding of the Rule Engine Tracing user interface along with how to use the feature to troubleshoot issues.

     

     

    Explanation of Rule Engine Tracing UI

     

    The easiest way to demonstrate the power or Rule Engine Tracing is with an example.

     

    Example 1: A website loads but without images---only text is displayed

     

    A user goes to cnn.com while going through the Web Gateway. The page loads, but it does not have any images and only text is being displayed. It looks like the image below.

     

     

    CNN+not+working.png

     

     


    Let's reproduce the problem while using Rule Tracing Central. In the Web Gateway UI, go to the Troubleshooting,
    Rule Tracing Central section.  Use the screenshot below to follow along.

    Overview-Update2.png

     

     

    1. Trace your client: Enter the client IP from where you’ll replicate the issue above.  If using Central Mgmt, be sure to select the Web Gateway appliance through which the client traffic will be proxied.  Click ‘Go’ to start tracing.  On your client machine, refresh your browser to replicate the issue.  As your client's requests hit the Web Gateway, trace files will appear in the Traces pane (section 2). Note: Sometimes it can take a while before traces show up, especially on larger transactions---be patient. Stop the trace after the issue was replicated.

    2. Select the trace file: On the left-hand side, icons indicate the action that was taken for each request captured. These icons are the same icons that are used for the block, continue, authenticate, redirect, stop rule set, stop cycle and remove actions in your Web Gateway policy.

      In this example, you can see that several requests for cdn.turner.com/cnn/ are getting blocked. Most likely these are from the content server for CNN. Click on one of the blocked traces to populate the right-side panes and find out why it is getting blocked.

    3. Analyze trace result:The upper right-hand pane gets populated with your Web Gateway policy tree view.  The view automatically jumps to the last action that was executed for this particular request. In our example that was a block action on the rule named "Block URLs Whose Category is in Category BlockList".

    4. Expand Details: The Details tab allows you to see evaluated properties,actions, and events associated with the selected rule.  In this example, we see the requests for cdn.turner.com/cnn/ were blocked as “Business” because this category is in the “Category Blocklist”.

      The Top Properties tab can also give you useful debugging information regarding a specific web request. In the example below, you can quickly identify the URL, URL.Host, and URL.Categories properties for whitelisting purposes.

      Top10-2.png

     

     

    Summary: Rule Engine Tracing quickly showed us that the URL, cdn.turner.com/cnn/*, is getting blocked by the rule "Block URLs whose Category is in Category Blocklist" because its category "Business" is in the list "Category Blocklist". To resolve the issue, you can create a whitelist rule by using the information contained in the Top Properties or Details tab. For information on whitelisting, click here.

     


    Example 2: User Group policy not applied as expected.

    A user receives an unexpected block page when attempting to access www.yahoo.comThe user should have been allowed access based on the expected policy assignment.  The user is supposed to be assigned the policy contained in the rule set "VIP" because of matched group membership criteria.

     

    yahooblock.png

     

     


    Let's reproduce the problem while using Rule Tracing Central. In the Web Gateway UI, go to the TroubleShooting,
    Rule Tracing Central section.  Use the screenshot below to follow along.

    Overview-not+working+trace.png

     

     

    1. Trace your client: Enter the client IP from where you’ll replicate the issue above.  If using Central Mgmt, be sure to select the Web Gateway appliance through which the client traffic will be proxied.  Click ‘Go’ to start tracing. On your client machine, refresh your browser to replicate the issue.  As your client's requests hit the Web Gateway, trace files will appear in the Traces pane (section 2). Note: Sometimes it can take a while before traces show up, especially on larger transactions---be patient. Stop the trace after the issue was replicated.

    2. Select the trace file: On the left-hand side, select the appropriate trace file. Since we were blocked when requesting www.yahoo.com, we will choose the trace file for the request with a block icon next to it.

    3. Analyze Trace results: The upper right-hand pane gets populated with your Web Gateway policy tree view.  The view automatically jumps to the last action that was executed for this particular request. In our example that was a block action in the rule "Block URLs Whose Category is in Category BlockList" inside of the ruleset "URL Filtering".

    4. Expand Details: The Details tab allows you to see evaluated properties, actions, and events associated with the selected rule.  In this example, we see the requests for www.yahoo.com were blocked as “Portal Sites” because this category is in the “Category Blocklist”.

     

     

    The results we found were unexpected as the client's request should have reached the "VIP" ruleset as opposed to the "URL Filtering" ruleset which restricted access to this site. Next, we'll want to focus our attention to the "VIP" ruleset to figure out why the request did not match its criteria. Use the screenshot below to follow along.

     

    VIP.png

     

     

    1. Select the VIP ruleset: Note that the arrow next to "VIP" is hollow. This indicates that the request did not match the criteria to go inside of that ruleset.
    2. Analyze Criteria: Focus on the 'Details' pane to analyze the criteria.
          a. Criteria: This is the criteria you've added for the request to enter the "VIP" ruleset. According to your rules, a user must be a member of the group "VIPo". ("Authentication.UserGroups contains "VIPo")
          b. Evaluated Criteria: This is the results of the Authentication.UserGroups property being filled. The user is part of the "VIP" group; not the "VIPo" group -  indicating that the admin entered a typo into the criteria---whoops!!.

     

    Summary: Rule Engine Tracing quickly showed us that the user's request didn't match the expected "VIP" rule due to a typo in the defined criteria.  The easy solution is to update the criteria to match the actual group name. 

     

     

     

     

    Example 3: Getting blocked by embedded content unexpectedly.

     

    A user receives an unexpected block page when attempting to download a zip file. The administrator expected the zip file to download as the admin does not block zip files in their Media Type Blocklist rule.

     

    3.png

     


    Let's reproduce the problem while using Rule Tracing Central. In the Web Gateway UI, go to the TroubleShooting,
    Rule Tracing Central section.  Use the screenshot below to follow along.

     

     

    1-modified.png

     

     

    1. Trace your client: Enter the client IP from where you’ll replicate the issue above.  If using Central Mgmt, be sure to select the Web Gateway appliance through which the client traffic will be proxied.  Click ‘Go’ to start tracing.  On your client machine, refresh your browser to replicate the issue.  As your client's requests hit the Web Gateway, trace files will appear in the Traces pane (section 2). Note: Sometimes it can take a while before traces show up, especially on larger transactions---be patient. Stop the trace after the issue was replicated.

    2. Select the trace file: On the left-hand side, icons indicate the action that was taken for each request captured. In this example, we know that we were blocked, so we select the trace file associated with the 'block' icon.

    3. Analyze trace result:The upper right-hand pane gets populated with your Web Gateway policy tree view.  The view automatically jumps to the last action that was executed for this particular request. In this example, that was a block action on the rule named "Block Types from Download Media Type Blocklist".

      Notice the highlighted red arrow pointing to the left and the number '2' in the filled rectangle. This indicates:

      • The request was blocked in the response's embedded cycle.
      • The composite opener is enabled in the policy/rules. Calling the composite opener causes the MWG to start extracting the Zip file -  sending all archive members as individual embedded cycles through the rule engine.
      • The '2' indicates that two embedded cycles were processed before the block action was triggered.

    4. Expand Details: The Details tab allows you to see evaluated properties, actions, and events associated with the selected rule.  Let's focus on why this rule is blocking the download of the zip file. In this example, you can see that two archive members of the zip file were extracted and sent as embedded cyles through the rule engine.

      Response embedded 1 shows that the Media Type Filtering rules detects the first archive as 'application/ms-html-help'. Notice the grey 'X'  next to the Criteria in this first embedded cycle. This indicates that the criteria of the rule was evaluated to be false. (not matched)

      Response embedded 2 shows that the Media Type Filtering rules detects the second archive as 'application/exe'. Notice the grey check-mark  next to the Criteria in this second cycle. This indicates that the criteria evaluated to a true statement and a block action was applied.

      Summary: Rule Engine tracing showed us that it was the embedded archives inside the zip file that triggered the block page. MWG's composite opener extracted the zip file and sent all of the archive members to the rule engine for inspection. One of the archive members was an executable, which was a media type that this Administrator was blocking. If this particular download was needed, the Administrator could add an whitelist entry for this particular URL. For information on whitelisting, you can find more information here: https://community.mcafee.com/docs/DOC-4514

     



     

     

     

    Additional Information


    Top Properties

    The Top Properties tab can show an administrator very useful debug information. This section shows a very clean view of the most commonly used properties and their values for each web request. The top properties can be a benefit for troubleshooting issues and for building rules in your policy.

     

     

    For instance, if you wanted to apply a rule based on the client's User-Agent, you're very quickly able to see what User-Agent the client is using. For more information on User-Agents, click here.

     

    Another example could be that you're looking to whitelist a specific URL.Host, but you're not sure exactly what value is filled for that property. The Top properties is the quickest means to find that information. Use the Top Properties instead of sifting through your access log or through a wireshark trace.  For more information on whitelisting URLs, click here.

     

     

     

    4-top-properties.png

    Cycles

    The cycles pane is useful when you are trying to figure out when a rule or ruleset was processed and the outcome of the rule. Cycles are represented by arrows in different colors. The meanings of each arrow are as follows:

        • Arrow pointing to the right -- Request Cycle
        • Arrow pointing to the left -- Response Cycle
        • No arrow pointing to the right(left) -- No processing in request(response) cycle.
        • Hollow Arrow -- Rule or Rule Set was processed but the criteria was not matched. No Action/Event took place.
        • Gray Arrow -- Action executed, but not as the most impacting action.
        • Green Arrow -- Stop Rule Set, Stop Cycle, or Continue executed as the most impacting action in trace.
        • Yellow Arrow -- Remove executed as most impacting.
        • Blue Arrow -- Authenticate executed as most impacting.
        • Dark Green Arrow -- Redirect executed as most impacting.
        • Red Arrow -- Block executed as most impacting.

     

     

    Below are a few examples of each cycle that we can look into to get further information. For this example we will examine the processing of an EXE inside of an archive file.

     

    When we first select the trace it will have all of the cycles selected. We can use the drop down bar to dig deeper into each cycle.

    allcycles.png

     

    The Link to List feature will allow you to modify list entries in your policy. This allows the administrator to both troubleshoot and fix your problem within the Rule Tracing Central interface.  

     

    Let's assume your block for a specific URL is not functioning, below is an example of how the Link to List can be used to quickly resolve issues.

     

    Our rule to block this URL is: URL (is in list) Block domains.

     

    Currently there is just the one entry in the list:

     

    BadEntry.png

     

     

    We can see here that the URL being requested is not "bandcamp.com" but http://jesu.bandcamp.com. To correct the list entry click the link to "Open current, local list". this will open the Edit List (String) window:

     

    HiLightedLink.jpg

     

    You can now proceed to change the entry from bandcamp.com to jesu.bandcamp.com. Once the edit is complete Save Changes and test again.

     

    BadEntry.png

     

     

    modifiedRule.JPG

     

     

    Another rule trace shows the URL requested being blocked as desired.

     

    BlockedProperly.JPG

     

     

     

    Filters

     

    With the new Rule Tracing Central, we can also filter the traces down to find the exact ones we are looking for. Below is an image of the available filters.

     

    2-availablefilters.png

     

     

    We can use these filters to find exact traces. For example, I want to see traces which major actions are either Block, Authenticate, or Continue and contains yahoo.com in the URL.

     

    3-filtered.png

     

     

     

    Import/Export

    Rule Tracing Central also features the ability to import and export rule traces from the GUI. When you export selected/visible traces, you will be able to save them in a zip archive. This can prove useful for storing old traces or to submit traces to Web Gateway Support. You can also import traces from other Web Gateways or from prior troubleshooting attempts.

     

     

    5-export.png

     

     

     

    Deleting Rule traces

    Rule engine traces can be removed from the panes of Rule Tracing Central but they still exist on the filesystem of the appliance. To delete the rule traces from the filesystem you need to access the rule tracing files section, which can be found under the <appliance> of the Troubleshooting menu.

     

     

    6-Rule+Tracing+Files.png

     

     

     

    If you  leave rule engine tracing on by accident, you shouldn't worry about the appliance filling up as we only store up to 5000 traces. When we go over  that number, it begins deleting the oldest trace. Deletion is not reflected in the Rule Engine UI, so you might see entries you cannot  access because they have been deleted.