cancel
Showing results for 
Search instead for 
Did you mean: 

Web Gateway: Troubleshooting with Rule Engine Tracing

Web Gateway: Troubleshooting with Rule Engine Tracing

What is Rule Engine Tracing?

Rule Engine Tracing is a visual representation that a request, response, and embedded cycles take through your policy. Rule Tracing Central was designed to make debugging your rules easy, help you understand the flow a web request takes through the rule engine, and save you time!

Have you ever needed to find out why a site was getting blocked or why your whitelist isn't working correctly? Did you ever have to figure out why a user was not assigned to the correct rules in your Policy? Rule Engine Tracing will help you solve all of these problems.

In this document you will gain a basic understanding of the Rule Engine Tracing user interface along with how to use the feature to troubleshoot issues.

Rule Engine Tracing UI Overview

The easiest way to demonstrate the power of Rule Engine Tracing is with an example.

Example 1: A website loads but without images - only text is displayed

A user goes to cnn.com while going through the Web Gateway. The page loads, but it does not have any images and only text is being displayed. It looks like the image below.

Let's reproduce the problem while using Rule Tracing Central. In the Web Gateway UI, go to the Troubleshooting > Rule Tracing Central section. Use the screenshot below to follow along.

Overview-Update2.png

1. Trace your client: Enter the client IP from where you’ll replicate the issue above.  If using Central Mgmt, be sure to select the Web Gateway appliance through which the client traffic will be proxied.  Click ‘Go’ to start tracing.  On your client machine, refresh your browser to replicate the issue.  As your client's requests hit the Web Gateway, trace files will appear in the Traces pane (section 2). Note: Sometimes it can take a while before traces show up, especially on larger transactions---be patient. Stop the trace after the issue was replicated.

2. Select the trace file: On the left-hand side, icons indicate the action that was taken for each request captured. These icons are the same icons that are used for the block, continue, authenticate, redirect, stop rule set, stop cycle and remove actions in your Web Gateway policy.

In this example, you can see that several requests for cdn.turner.com/cnn/ are getting blocked. Most likely these are from the content server for CNN. Click on one of the blocked traces to populate the right-side panes and find out why it is getting blocked.

3. Analyze trace result:The upper right-hand pane gets populated with your Web Gateway policy tree view.  The view automatically jumps to the last action that was executed for this particular request. In our example that was a block action on the rule named "Block URLs Whose Category is in Category BlockList".

4. Expand Details: The Details tab allows you to see evaluated properties,actions, and events associated with the selected rule.  In this example, we see the requests for cdn.turner.com/cnn/ were blocked as “Business” because this category is in the “Category Blocklist”.

The Top Properties tab can also give you useful debugging information regarding a specific web request. In the example below, you can quickly identify the URL, URL.Host, and URL.Categories properties for whitelisting purposes.

Top10-2.png

Summary: Rule Engine Tracing quickly showed us that the URL, cdn.turner.com/cnn/*, is getting blocked by the rule "Block URLs whose Category is in Category Blocklist" because its category "Business" is in the list "Category Blocklist". To resolve the issue, you can create a whitelist rule by using the information contained in the Top Properties or Details tab. For information on whitelisting, click here.

Example 2: User Group policy not applied as expected

A user receives an unexpected block page when attempting to access www.yahoo.com.  The user should have been allowed access based on the expected policy assignment.  The user is supposed to be assigned the policy contained in the rule set "VIP" because of matched group membership criteria.

Let's reproduce the problem while using Rule Tracing Central. In the Web Gateway UI, go to the Troubleshooting > Rule Tracing Central section.  Use the screenshot below to follow along.

1. Trace your client: Enter the client IP from where you’ll replicate the issue above.  If using Central Mgmt, be sure to select the Web Gateway appliance through which the client traffic will be proxied.  Click ‘Go’ to start tracing. On your client machine, refresh your browser to replicate the issue.  As your client's requests hit the Web Gateway, trace files will appear in the Traces pane (section 2). Note: Sometimes it can take a while before traces show up, especially on larger transactions---be patient. Stop the trace after the issue was replicated.

2. Select the trace file: On the left-hand side, select the appropriate trace file. Since we were blocked when requesting www.yahoo.com, we will choose the trace file for the request with a block icon next to it.

3. Analyze Trace results: The upper right-hand pane gets populated with your Web Gateway policy tree view.  The view automatically jumps to the last action that was executed for this particular request. In our example that was a block action in the rule "Block URLs Whose Category is in Category Blocklist" inside of the ruleset "URL Filtering".

4. Expand Details: The Details tab allows you to see evaluated properties, actions, and events associated with the selected rule.  In this example, we see the requests for www.yahoo.com were blocked as "Portal Sites" because this category is in the "Category Blocklist".

The results we found were unexpected as the client's request should have reached the "VIP" ruleset as opposed to the "URL Filtering" ruleset which restricted access to this site. Next, we'll want to focus our attention to the "VIP" ruleset to figure out why the request did not match its criteria. Use the screenshot below to follow along.

1. Select the VIP ruleset: Note that the arrow next to "VIP" is hollow. This indicates that the request did not match the criteria to go inside of that ruleset.

2. Analyze Criteria: Focus on the 'Details' pane to analyze the criteria.

    a. Criteria: This is the criteria you've added for the request to enter the "VIP" ruleset. According to your rules, a user must be a member of the group "VIPo". ("Authentication.UserGroups contains "VIPo")

    b. Evaluated Criteria: This is the results of the Authentication.UserGroups property being filled. The user is part of the "VIP" group; not the "VIPo" group -  indicating that the admin entered a typo into the criteria---whoops!!.

Summary: Rule Engine Tracing quickly showed us that the user's request didn't match the expected "VIP" rule due to a typo in the defined criteria.  The easy solution is to update the criteria to match the actual group name.

Example 3: Getting blocked by embedded content unexpectedly

A user receives an unexpected block page when attempting to download a zip file. The administrator expected the zip file to download as the admin does not block zip files in their Media Type Blocklist rule.

Let's reproduce the problem while using Rule Tracing Central. In the Web Gateway UI, go to the Troubleshooting > Rule Tracing Central section.  Use the screenshot below to follow along.

1. Trace your client: Enter the client IP from where you’ll replicate the issue above.  If using Central Mgmt, be sure to select the Web Gateway appliance through which the client traffic will be proxied.  Click ‘Go’ to start tracing.  On your client machine, refresh your browser to replicate the issue.  As your client's requests hit the Web Gateway, trace files will appear in the Traces pane (section 2). Note: Sometimes it can take a while before traces show up, especially on larger transactions---be patient. Stop the trace after the issue was replicated.

2. Select the trace file: On the left-hand side, icons indicate the action that was taken for each request captured. In this example, we know that we were blocked, so we select the trace file associated with the 'block' icon.

3. Analyze trace result: The upper right-hand pane gets populated with your Web Gateway policy tree view.  The view automatically jumps to the last action that was executed for this particular request. In this example, that was a block action on the rule named "Block Types from Download Media Type Blocklist".

Notice the highlighted red arrow pointing to the left and the number '2' in the filled rectangle. This indicates:

        • The request was blocked in the response's embedded cycle.
        • The composite opener is enabled in the policy/rules. Calling the composite opener causes the MWG to start extracting the Zip file -  sending all archive members as individual embedded cycles through the rule engine.
        • The '2' indicates that two embedded cycles were processed before the block action was triggered.

4. Expand Details: The Details tab allows you to see evaluated properties, actions, and events associated with the selected rule.  Let's focus on why this rule is blocking the download of the zip file. In this example, you can see that two archive members of the zip file were extracted and sent as embedded cycles through the rule engine.

Response embedded 1 shows that the Media Type Filtering rules detects the first archive as 'application/ms-html-help'. Notice the grey 'X'  next to the Criteria in this first embedded cycle. This indicates that the criteria of the rule was evaluated to be false. (not matched)

Response embedded 2 shows that the Media Type Filtering rules detects the second archive as 'application/exe'. Notice the grey check-mark  next to the Criteria in this second cycle. This indicates that the criteria evaluated to a true statement and a block action was applied.

Summary: Rule Engine tracing showed us that it was the embedded archives inside the zip file that triggered the block page. Web Gateway's composite opener extracted the zip file and sent all of the archive members to the rule engine for inspection. One of the archive members was an executable, which was a media type that this Administrator was blocking. If this particular download was needed, the Administrator could add an whitelist entry for this particular URL. For information on whitelisting, you can find more information here:

Additional Information

Top Properties

The Top Properties tab can show an administrator very useful debug information. This section shows a very clean view of the most commonly used properties and their values for each web request. The top properties can be a benefit for troubleshooting issues and for building rules in your policy.

For instance, if you wanted to apply a rule based on the client's User-Agent, you're very quickly able to see what User-Agent the client is using. For more information on User-Agents, click here.

Another example could be that you're looking to whitelist a specific URL.Host, but you're not sure exactly what value is filled for that property. The Top properties is the quickest means to find that information. Use the Top Properties instead of sifting through your access log or through a wireshark trace.  For more information on whitelisting URLs, click here:

Cycles

The cycles pane is useful when you are trying to figure out when a rule or ruleset was processed and the outcome of the rule. Cycles are represented by arrows in different colors. The meanings of each arrow are as follows:

    • Arrow pointing to the right -- Request Cycle
    • Arrow pointing to the left -- Response Cycle
    • No arrow pointing to the right(left) -- No processing in request(response) cycle.
    • Hollow Arrow -- Rule or Rule Set was processed but the criteria was not matched. No Action/Event took place.
    • Gray Arrow -- Action executed, but not as the most impacting action.
    • Green Arrow -- Stop Rule Set, Stop Cycle, or Continue executed as the most impacting action in trace.
    • Yellow Arrow -- Remove executed as most impacting.
    • Blue Arrow -- Authenticate executed as most impacting.
    • Dark Green Arrow -- Redirect executed as most impacting.
    • Red Arrow -- Block executed as most impacting.

Below are a few examples of each cycle that we can look into to get further information. For this example we will examine the processing of an EXE inside of an archive file. When we first select the trace it will have all of the cycles selected. We can use the drop down bar to dig deeper into each cycle.

Link to List

The Link to List feature will allow you to modify list entries in your policy. This allows the administrator to both troubleshoot and fix your problem within the Rule Tracing Central interface.

Let's assume your block for a specific URL is not functioning, below is an example of how the Link to List can be used to quickly resolve issues.

Our rule to block this URL is: URL (is in list) "Block domains". Currently there is just the one entry in the list:

We can see here that the URL being requested is not "bandcamp.com" but http://jesu.bandcamp.com. To correct the list entry click the link to "Open current, local list". this will open the Edit List (String) window:

You can now proceed to change the entry from bandcamp.com to jesu.bandcamp.com. Once the edit is complete Save Changes and test again.

Another rule trace shows the URL requested being blocked as desired.

Filters

With the Rule Tracing Central, we can also filter the traces down to find the exact ones we are looking for. Below is an image of the available filters.

We can use these filters to find exact traces. For example, I want to see traces which major actions are either Block, Authenticate, or Continue and contains yahoo.com in the URL.

Import/Export

Rule Tracing Central also features the ability to import and export rule traces from the GUI. When you export selected/visible traces, you will be able to save them in a zip archive. This can prove useful for storing old traces or to submit traces to Web Gateway Support. You can also import traces from other Web Gateways or from prior troubleshooting attempts.

Deleting Rule traces

Rule engine traces can be removed from the panes of Rule Tracing Central but they still exist on the filesystem of the appliance. To delete the rule traces from the filesystem you need to access the rule tracing files section, which can be found under the <appliance> of the Troubleshooting menu.

If you  leave rule engine tracing on by accident, you shouldn't worry about the appliance filling up as we only store up to 5000 traces. When we go over  that number, it begins deleting the oldest trace. Deletion is not reflected in the Rule Engine UI, so you might see entries you cannot  access because they have been deleted.

Comments
Regis

This GUI for digesting rule tracing output is awesome.    Thanks to all who worked together to make this so useful.   

phlrnnr

I love Rule Tracing Central, and this is a great document describing its usefulness!  I have one question about the cycles pane.  Sometimes in the cycles pane there will be a number.  For example, from your screenshot above:

cycle.JPG

What does this number represent, and how can that help in troubleshooting a problem or understanding what the Web Gateway is doing?

Thanks!

sgletenko

I have MWG 4500 appliance, version 7.4.0.

It makes 430 000 traces, after this free space on /opt ends (40 Gb, default disk partitioning).

I see them in ssh shell, they dont delete automatically.

How to allow only 5000 traces ?

dshock

Hi phlrnnr -

We updated the article to address the question you posed in regards to what the number represents. Check out Example 3 and let us know if you have additional questions!

-Darin

phlrnnr

Darin, thanks!  That clarification is very helpful!

haaris

How to trace for multiple IPs in Rule tracing??

Is there any way to trace for multiple IPs or we can trace only single IP at one time

dshock

Hey - you can use rule traces using the manual method.

Please download the following rule tracing ruleset file:

ftp://ftp.support.securecomputing.com/outgoing/troubleshooting.zip

   1. Extract and import it using the ruleset library (Policy > Rule Sets > Add > Rule Set from Library... > Import from File)

   2. Place this ruleset at the top of your ruleset

   3. Add your client IP to the "Rule Engine Tracing List"

   4. Enable the rule (Enable Rule Engine Tracing for IPs in Rule Engine Tracing List)

   5. Reproduce the issue

   6. Disable the rule (Enable Rule Engine Tracing for IPs in Rule Engine Tracing List)

   7. Locate the rule trace files under Troubleshooting > Rule Tracing Files. You can select the individual files and then analyze them in the Rule Engine Tracing UI.

haaris

Thanks for the update.

It really works

aaadam

A yellow triangle with exclamation mark appears sometimes in rule tracing log. Please what does it mean? There is no context help on mouse over.

triangle.png

chris_lee

Hi,

how about the Request 2 mean in the trace there. I see how traffic have request have "request 2" some don't. and both is HTTPS traffic.

Thank you.

Teh

johnaldridge

Request 2 only occurs during SS/TLS connections.

You will notice that the HTTP method used will appear as:

Command.NameCERTVERIFY

And, notice that the actual request is a CONNECT, which also has no response cycle, no embedded content, and no path.  These are easily recognized in the trace listing as having no trailing slash.

Version history
Revision #:
1 of 1
Last update:
‎08-12-2013 11:30 AM
Updated by: