cancel
Showing results for 
Search instead for 
Did you mean: 

SIEM Foundations: Filter Out Low-Value Events

SIEM Foundations: Filter Out Low-Value Events

Some data sources that you have reporting into your SIEM may be very noisy, and not all the data they contribute may be useful in the context of your SIEM use cases.  As an example, low level TCP setup/teardown events that are generated by some firewalls and other network gear tend not to be very useful, but can often represent a high volume of events. By eliminating high-volume, low-value events you can streamline your SIEM operations, and ensure you are achieving the highest performance possible.

Events are filtered via Filter Rules, which you’ll find in the Policy Editor.

policy-editor.png

Policy Editor


Filter rules are located under the Receiver policy:

filter-rules.png


Filter rules leverage string matches and/or regular expressions to identify which events to match against.  Filter rules are executed before event parsing, and so can be an effective way to improve Receiver performance in environments where you have devices sending high volumes of irrelevant events. Filter rules have a set of selectable actions to identify what actions to take when an event matches a rule.

As an example, we’ll create a rule that eliminates UDP teardown events generated by our Cisco PIX firewall.  From the screenshot below, you’ll see that we have millions of these unwanted events in the SIEM:

pic1.png

We’d like to implement a filter rule to eliminate the UDP teardown events from being parsed and set to the ESM, while still maintaining them in the Enterprise Log Manager for long-term compliance reporting needs.

To filter unwanted events:

  • Identify a string or regular expression that properly matches your set of unwanted events.  Since filtering happens before parsing, you will not be able to match on specific parsed fields of an event.

Our UDP teardown logs look like this (as seen in the Packet tab of the Event Details view):

Jul 31 2014 20:49:04: %PIX-6-302016: Teardown UDP connection 84966377 for outside:203.144.255.72/53 to dmz:10.0.171.165/32793 duration 0:00:01 bytes 281

  • Create a new Filter Rule with your expression.  In the filter policy, select New/Filter Rule.  There are two main parts to the rule.  The first group of settings identify what events will match the rule, and the second identifies what to do with events that match our rule.  There are two different ways to match an event to our rule.


    • Content Strings.  Content Strings are simple static string matches.  If your unwanted events have an easily-identifiable static string that you can use to identify them, then this is where you should put it.  In our example events, the string


%PIX-6-302016: Teardown UDP connection


is seen consistently in all the unwanted events, and we’ll use this as our simple match string.


    • PCRE (Perl Compatible Regular Expression):  PCRE supports complex expressions that you can use to identify events you'd like to filter.  If we configure both a Content String and PCRE, the PCRE is only applied against events that match the Content String first.  Content Strings can be used as an effective first-level filter to ensure that the more expensive PCRE operation is only applied to events that pass through the Content String filter.  Our string is a simple match, so does not require a regular expression.  The simple Content String match will do.

pic2.png

The second part of the Filter rule is defining what to do when the filter rule matches.  Available actions include:

    • Send Log to ParserThis lets the events matching the expression continue on its usual way to the parser.
    • Send Log to ELMForward the events matching the expression to the ELM.
    • Generate ESM EventThis will directly create an appropriate internal event on the ESM each time this filter rule matches an event. Depending on the type of logs, that can create a very high number of events. Use very carefully.
    • Stop Processing Filter Rules. Does not process any more filter rules after this matched.

In our example, we will choose only to send the log to the ELM, and stop processing additional rules for any events that match the content string.

  • Enable rules and roll out policy.  Finally, ensure that your new filter rules are enabled in the Policy Editor, and then roll out policy by selecting Operations/Rollout.  From this point forward, we should see no more of these pesky UDP teardown events.

« previousoutlinenext »

Comments
rhinomike

the filtering UI in ESM is a nearly a crime against humanity...

99% of the time all a person wants is to filter chatty log entries out of ESM while sending them to ELM...

As such, usability would be greatly improved if the parser window had an extra action called "ELM only".

staschler

While I agree the filtering UI has a lot of room to improve, it's worth pointing out that there's a reason for it.  Filtering on the Receiver happens before parsing, and the feature is there mostly to eliminate the extra workload associated with parsing events you don't care about.  Since the REC hasn't parsed the log yet when it hits the filtering engine, it has limited information to go on.  In general it wouldn't work to perform event filtering in the policy for the parser rule.  One parser rule often deals with many different types of events, and the REC doesn't know which event is has until the event is fully parsed.  By that point, filter rules are irrelevant.

If you have an entire data source that you don't want to parse, and only send to ELM, then you can set this with the "logging/parsing" check boxes in the data source config.

If you need to get specific, and send some, but not all logs from a given data source to the ESM, then you have 2 options:

  1. Disable the data source rules associated with the events you don't want to see in the ESM.  All events will still go to the ELM (assuming you have logging enabled in the data source config).  Your REC will still parse the unwanted events, so you haven't gained any performance advantage.  It will simply discard any that get mapped to disabled data source rules in the data source policy.  If your REC is not performance constrained, then this is a perfectly acceptable option, and eliminates the need to monkey with filtering rules.
  2. If you are in a position where you would like to reduce the load on your REC, then we need a way to selectively define the desired flow for logs that have been collected, without parsing them.  That's what filtering rules are there for.  The filter pattern matching engine is more efficient than the full-blown parser, but less functional.

Scott

rhinomike

Scott,

First thank you for the reply. Really great to see experience McAfee staff engaging users directly (rather than the relying on unskilled moderators).

Regarding option 1: When you write


"Disable the data source rules associated with the events you don't want to see in the ESM"



Are you referring to that the "Enabled" line below is to be set for both Parsing and Logging?


Add-Data-Source---SYSLOGcircle.png

Followed by navigating to Data Source Rules of an specific source and choosing for example, rule ID 256-1012387 and setting action to Disabled?

If not, would you be able to document what exactly you refer to?

staschler

Yes, Mike, that's exactly what I'm proposing. 

Scott

mcgarl1

Scott,

Can the same "filtering" be accomplished by the following? Basically I found some domain controller events that are of no concern. Can't we just go into the Policy Editor and disable them as shown in the pic below?

Policy Editor.PNG

staschler

Yes, if the events you're looking to filter map directly to a single Windows rule or Data Source rule, then disabling that rule is a fine way to filter.  It's easier than using filtering rules, but slightly less efficient.  In this case, the Rec needs to parse the collected logs in order to map them to the rule.  Then, when it sees the rule is disabled, it will discard it.  In the case of filtering rules, the filtering happens before parsing, so you don't waste that effort.  If you're talking about relatively low-volume events, this impact is negligible.  However, if you're looking at very high volumes of filtered events, this can make a lot of difference in performance at the receiver.

Scott

mcgarl1

Got it! I am still a little confused with the "Stop Processing Filter Rules" being checked.

  • When this option is checked, what happens to the future events that come through containing the string? Do they no longer get filtered?


Thanks,

LT

jon286

It reads to me it won't run the filter rules following this one once the event matches.

jon286

Are the filters run in such a way they support multiple content strings? In my case to filter out the root server dests. from a given event. The strings will work individually, but I still see the events with this filter rule, the 78 strings I'd need doesn't sound particularly efficient but I'd otherwise need 78 filter rules.

contentstrings.jpg

rgarrett

Just noticed your comment.. Content strings are ANDed. Thus the filter engine would look for an event that matched all of those.  Probably not what you wanted.  Best to separate them.

jon286

I'd thought as much; in the end I could do what was far simpler as I was able to put those through their own firewall rule, and just disable syslog on that rule.

minki

Hi, Could you please help in below points:

1> How can I create a filter for only specific receivers.

     For example, we have 4 receivers and I just want to apply the filter only to three receivers within the same filter where the content string is same for all.

2> We have couple of firewalls from different vendors because of that "Tear Down" events content string is different but in all type of "Team down" events we can see "Tear Down" string regardless of the starting and ending of the string, in this case, how can I filter out the "Tear Down" events for all types of firewall in single filter.

3> Is there any limit on Filters.i.e. how many Filters we can create

auguste

Hello,

if i'm not wrong,
when a filter is created on a specific data source you must create a filter "ZZ_Catch_all" (to be the last filter rule) which sends logs to Parser, ESM, ELM, and stop processing filter rules. --> as explained in the KB74834

https://kc.mcafee.com/agent/index?page=content&id=KB74834

This is for accept all others events .

Version history
Revision #:
1 of 1
Last update:
‎08-10-2014 09:15 PM
Updated by: