9 Replies Latest reply on Oct 13, 2014 8:38 PM by btlyric

    Rule counters

    nsgmike

      My rule base is getting a little out of hand, I would like to set up counters so I can track which rules are being used the most so I can audit it after a couple weeks and clean up the least used rules. How would I go about doing this? Are there any counters built in that I can activate?

       

      Thank you

        • 1. Re: Rule counters

          There are no counters, but I can setup a rule usage log is used to import into policyViewer.

          If you create a log handler rule as described below, it creates one line entry with all the rules that have fired.

          Do not keep this rule on for any extended length of time, maybe a couple of hours. I don't know how large it could get on a busy system.

          Here's the ReadMe for it i include in policyViewer:

           

           

          policyViewer 1.4.0 introduces a Rule Usage feature.

           

           

          In order to understand which rules in your policy are actually being

          used and to better optimize your policy, you can create a log described

          below and import into policyViewer. policyViewer then displays the

          evaluated and fired hits for the request/response/embedded cycles in the

          Rule Sets tree view on the left pane and the policy output on the right

          pane.

           

          [evalReq(firedReq)/evalResp(firedResp)/evalEmbed(firedEmbed)]

           

          The RuleUsage.log must have the following minimum format:

           

          Log header:

          date time "evaluated" "fired"

           

          Log Handler:

          Set User-Defined.logLine =

               DateTime.Date.ToString("%YYYY-%MM-%DD") +

               " " +

               DateTime.Time.ToString("%hh:%mm:%ss") +

               " "" +

               List.OfString.ToString(Rules.EvaluatedRules) +

               "" "" +

               List.OfString.ToString(Rules.FiredRules) +

               """

          FileSystemLogging.WriteLogEntry(User-Defined.logLine)<RuleUsage.log>

           

          Rules.EvaluatedRules is a list of rule IDs that the request walked

          through in the policy. Rules.FiredRules is a list of rule IDs that

          actually triggered as true and performed that rule's action.

           

          When you load the .backup or feedback of the exact same policy,

          right-click on a ruleset in the tree view and "Import Rule Usage Log".

          (This will not work with imported Rule Sets.)

           

          Select the RuleUsage.log(s) that have been downloaded from MWG and

          stored locally. The logs must be decompressed before import. (Support

          for .gz logs in future version.)

           

          WARNING: Only keep RuleUsage.log enabled to record requests for a very

          short period of time. It could log a large amount of data, depending on

          the number of rules in your policy and number of requests recorded

          during the period.

           

           

           

          As usual, this is not supported by McAfee, so use with some discretion.

           

          • 2. Re: Rule counters
            nsgmike

            Thank you eric, I just built this log handler, enabled it for about 5 minutes but I did not see any log files created yet? Something I am missing?

             

            ruleusage.JPG

            • 3. Re: Re: Rule counters

              Not quite.

              Import the attached rule set into the Default Log Handler.

              • 4. Re: Rule counters
                nsgmike

                The import failed, said something about the version and then I received a java error that would not let it complete, I am running 7.3.2.8.

                 

                Do you have a screenshot?

                • 5. Re: Rule counters

                  Just edit the raw XML file and replace the version tag with:

                  <version>7.3.2.8.0-17286</version>

                  or type this in:

                   

                  Set User-Defined.logLine =

                       DateTime.Date.ToString("%YYYY-%MM-%DD") +

                       " " +

                       DateTime.Time.ToString("%hh:%mm:%ss") +

                       " "" +

                       List.OfString.ToString(Rules.EvaluatedRules,", ") +

                       "" "" +

                       List.OfString.ToString(Rules.FiredRules,", ") +

                       """

                  • 6. Re: Rule counters
                    btlyric

                    e2 -- your method is definitely less pain on the configuration side, but would the following approach also work and potentially be less resource intensive?

                     

                    Create user defined statistics counters via Settings -> Statistics -> (configured instance) -> Statistics User Defined Counters

                     

                    Examples:

                     

                    Name     Type

                    Rule_1     Incremental

                    Rule_2     Incremental

                    Rule_3     Incremental

                     

                    Then, in the policy, in Rule 1, add the event Statistics.Counter.Increment("Rule_1",1)<Default> and so on, for each relevant rule.

                     

                    Then you could pull the /opt/mwg/lock/statistic/statistics.db file from each proxy and analyze the data manually.

                     

                    It's a pretty ugly solution, but theoretically it seems like it would work.

                    • 7. Re: Rule counters
                      yuems

                      Hi,

                       

                      I tried this and copy /opt/mwg/lock/statistic/statistics.db file.
                      I only open it with SQLite SQLite Download Page

                      And it contains 3 tables,
                      Chield
                      Stat

                      Version

                       

                      How can I count rule match?

                       

                      Kind Regards.

                      • 8. Re: Rule counters

                        Creating a counter for each rule you may want to track might be one way to go. It would be very tedious for every rule, but maybe just some key rules.

                        However, getting the data back out would be challenging if you are trying to access the statistics.db directly. The binary data blob with the value is in a encoded format that is used for dashboards, and not easily extractable.

                        If i were going to display those values, i would be more inclined to put them on a block page with Statistics.Counter.Get() statements.

                         

                        I've use the RuleUsage.log method a few times with customers to help optimize their policy. you can easily see sections of rules that never get hit, and you can visulaize a little better the flow of Request/Response/Embedded cycles. It's not perfect. but it's not bad.

                        • 9. Re: Rule counters
                          btlyric

                          Good thought on the block page with Statistics.Counter.Get(). I was thinking about implementing a custom dashboard, but the block page is probably easier. Checked it out a bit using Statistics.Counter.Increment(Rules.CurrentRule.Name,1)<Default>, pulled a list of rule names and did a regex replace against <b>Rule: $1: </b>$<propertyInstance useMostRecentConfiguration="false" configurationId="com.scur.engine.billing.4575" propertyId="com.scur.engine.billing.counter.get"><parameters><entry><string>com .scur.engine.billing.counter.get.name</string><parameter valueTyp="3"><value><stringValue value="$1.4575" stringModifier="true" typeId="com.scur.type.string"/></value></parameter></entry></parameters></prope rtyInstance>$<br /> to get the entries to populate the block page. Would be nice if it wasn't necessary to manually add the user-defined stats counters to the stats engine and/or there was some sort of simple way to generate rule usage statistics, either through MWG or through a 3rd party product.