cancel
Showing results for 
Search instead for 
Did you mean: 
nsgmike
Level 7

Rule counters

Jump to solution

My rule base is getting a little out of hand, I would like to set up counters so I can track which rules are being used the most so I can audit it after a couple weeks and clean up the least used rules. How would I go about doing this? Are there any counters built in that I can activate?

Thank you

1 Solution

Accepted Solutions
eelsasser
Level 15

Re: Rule counters

Jump to solution

There are no counters, but I can setup a rule usage log is used to import into policyViewer.

If you create a log handler rule as described below, it creates one line entry with all the rules that have fired.

Do not keep this rule on for any extended length of time, maybe a couple of hours. I don't know how large it could get on a busy system.

Here's the ReadMe for it i include in policyViewer:

policyViewer 1.4.0 introduces a Rule Usage feature.

In order to understand which rules in your policy are actually being

used and to better optimize your policy, you can create a log described

below and import into policyViewer. policyViewer then displays the

evaluated and fired hits for the request/response/embedded cycles in the

Rule Sets tree view on the left pane and the policy output on the right

pane.

[evalReq(firedReq)/evalResp(firedResp)/evalEmbed(firedEmbed)]

The RuleUsage.log must have the following minimum format:

Log header:

date time "evaluated" "fired"

Log Handler:

Set User-Defined.logLine =

     DateTime.Date.ToString("%YYYY-%MM-%DD") +

     " " +

     DateTime.Time.ToString("%hh:%mm:%ss") +

     " "" +

     List.OfString.ToString(Rules.EvaluatedRules) +

     "" "" +

     List.OfString.ToString(Rules.FiredRules) +

     """

FileSystemLogging.WriteLogEntry(User-Defined.logLine)<RuleUsage.log>

Rules.EvaluatedRules is a list of rule IDs that the request walked

through in the policy. Rules.FiredRules is a list of rule IDs that

actually triggered as true and performed that rule's action.

When you load the .backup or feedback of the exact same policy,

right-click on a ruleset in the tree view and "Import Rule Usage Log".

(This will not work with imported Rule Sets.)

Select the RuleUsage.log(s) that have been downloaded from MWG and

stored locally. The logs must be decompressed before import. (Support

for .gz logs in future version.)

WARNING: Only keep RuleUsage.log enabled to record requests for a very

short period of time. It could log a large amount of data, depending on

the number of rules in your policy and number of requests recorded

during the period.

As usual, this is not supported by McAfee, so use with some discretion.

9 Replies
eelsasser
Level 15

Re: Rule counters

Jump to solution

There are no counters, but I can setup a rule usage log is used to import into policyViewer.

If you create a log handler rule as described below, it creates one line entry with all the rules that have fired.

Do not keep this rule on for any extended length of time, maybe a couple of hours. I don't know how large it could get on a busy system.

Here's the ReadMe for it i include in policyViewer:

policyViewer 1.4.0 introduces a Rule Usage feature.

In order to understand which rules in your policy are actually being

used and to better optimize your policy, you can create a log described

below and import into policyViewer. policyViewer then displays the

evaluated and fired hits for the request/response/embedded cycles in the

Rule Sets tree view on the left pane and the policy output on the right

pane.

[evalReq(firedReq)/evalResp(firedResp)/evalEmbed(firedEmbed)]

The RuleUsage.log must have the following minimum format:

Log header:

date time "evaluated" "fired"

Log Handler:

Set User-Defined.logLine =

     DateTime.Date.ToString("%YYYY-%MM-%DD") +

     " " +

     DateTime.Time.ToString("%hh:%mm:%ss") +

     " "" +

     List.OfString.ToString(Rules.EvaluatedRules) +

     "" "" +

     List.OfString.ToString(Rules.FiredRules) +

     """

FileSystemLogging.WriteLogEntry(User-Defined.logLine)<RuleUsage.log>

Rules.EvaluatedRules is a list of rule IDs that the request walked

through in the policy. Rules.FiredRules is a list of rule IDs that

actually triggered as true and performed that rule's action.

When you load the .backup or feedback of the exact same policy,

right-click on a ruleset in the tree view and "Import Rule Usage Log".

(This will not work with imported Rule Sets.)

Select the RuleUsage.log(s) that have been downloaded from MWG and

stored locally. The logs must be decompressed before import. (Support

for .gz logs in future version.)

WARNING: Only keep RuleUsage.log enabled to record requests for a very

short period of time. It could log a large amount of data, depending on

the number of rules in your policy and number of requests recorded

during the period.

As usual, this is not supported by McAfee, so use with some discretion.

nsgmike
Level 7

Re: Rule counters

Jump to solution

Thank you eric, I just built this log handler, enabled it for about 5 minutes but I did not see any log files created yet? Something I am missing?

ruleusage.JPG

0 Kudos
eelsasser
Level 15

Re: Re: Rule counters

Jump to solution

Not quite.

Import the attached rule set into the Default Log Handler.

0 Kudos
nsgmike
Level 7

Re: Rule counters

Jump to solution

The import failed, said something about the version and then I received a java error that would not let it complete, I am running 7.3.2.8.

Do you have a screenshot?

0 Kudos
eelsasser
Level 15

Re: Rule counters

Jump to solution

Just edit the raw XML file and replace the version tag with:

<version>7.3.2.8.0-17286</version>

or type this in:

Set User-Defined.logLine =

     DateTime.Date.ToString("%YYYY-%MM-%DD") +

     " " +

     DateTime.Time.ToString("%hh:%mm:%ss") +

     " "" +

     List.OfString.ToString(Rules.EvaluatedRules,", ") +

     "" "" +

     List.OfString.ToString(Rules.FiredRules,", ") +

     """

0 Kudos
btlyric
Level 12

Re: Rule counters

Jump to solution

e2 -- your method is definitely less pain on the configuration side, but would the following approach also work and potentially be less resource intensive?

Create user defined statistics counters via Settings -> Statistics -> (configured instance) -> Statistics User Defined Counters

Examples:

Name     Type

Rule_1     Incremental

Rule_2     Incremental

Rule_3     Incremental

Then, in the policy, in Rule 1, add the event Statistics.Counter.Increment("Rule_1",1)<Default> and so on, for each relevant rule.

Then you could pull the /opt/mwg/lock/statistic/statistics.db file from each proxy and analyze the data manually.

It's a pretty ugly solution, but theoretically it seems like it would work.

yuems
Level 11

Re: Rule counters

Jump to solution

Hi,

I tried this and copy /opt/mwg/lock/statistic/statistics.db file.
I only open it with SQLite SQLite Download Page

And it contains 3 tables,
Chield
Stat

Version

How can I count rule match?

Kind Regards.

0 Kudos
eelsasser
Level 15

Re: Rule counters

Jump to solution

Creating a counter for each rule you may want to track might be one way to go. It would be very tedious for every rule, but maybe just some key rules.

However, getting the data back out would be challenging if you are trying to access the statistics.db directly. The binary data blob with the value is in a encoded format that is used for dashboards, and not easily extractable.

If i were going to display those values, i would be more inclined to put them on a block page with Statistics.Counter.Get() statements.

I've use the RuleUsage.log method a few times with customers to help optimize their policy. you can easily see sections of rules that never get hit, and you can visulaize a little better the flow of Request/Response/Embedded cycles. It's not perfect. but it's not bad.

Highlighted
btlyric
Level 12

Re: Rule counters

Jump to solution

Good thought on the block page with Statistics.Counter.Get(). I was thinking about implementing a custom dashboard, but the block page is probably easier. Checked it out a bit using Statistics.Counter.Increment(Rules.CurrentRule.Name,1)<Default>, pulled a list of rule names and did a regex replace against <b>Rule: $1: </b>$<propertyInstance useMostRecentConfiguration="false" configurationId="com.scur.engine.billing.4575" propertyId="com.scur.engine.billing.counter.get"><parameters><entry><string>com.scur.engine.billing.counter.get.name</string><parameter valueTyp="3"><value><stringValue value="$1.4575" stringModifier="true" typeId="com.scur.type.string"/></value></parameter></entry></parameters></propertyInstance>$<br /> to get the entries to populate the block page. Would be nice if it wasn't necessary to manually add the user-defined stats counters to the stats engine and/or there was some sort of simple way to generate rule usage statistics, either through MWG or through a 3rd party product.

0 Kudos