Some data sources that you have reporting into your SIEM may be very noisy, and not all the data they contribute may be useful in the context of your SIEM use cases. As an example, low level TCP setup/teardown events that are generated by some firewalls and other network gear tend not to be very useful, but can often represent a high volume of events. By eliminating high-volume, low-value events you can streamline your SIEM operations, and ensure you are achieving the highest performance possible.
Events are filtered via Filter Rules, which you’ll find in the Policy Editor.
Filter rules are located under the Receiver policy:
Filter rules leverage string matches and/or regular expressions to identify which events to match against. Filter rules are executed before event parsing, and so can be an effective way to improve Receiver performance in environments where you have devices sending high volumes of irrelevant events. Filter rules have a set of selectable actions to identify what actions to take when an event matches a rule.
As an example, we’ll create a rule that eliminates UDP teardown events generated by our Cisco PIX firewall. From the screenshot below, you’ll see that we have millions of these unwanted events in the SIEM:
We’d like to implement a filter rule to eliminate the UDP teardown events from being parsed and set to the ESM, while still maintaining them in the Enterprise Log Manager for long-term compliance reporting needs.
To filter unwanted events:
- Identify a string or regular expression that properly matches your set of unwanted events. Since filtering happens before parsing, you will not be able to match on specific parsed fields of an event.
Our UDP teardown logs look like this (as seen in the Packet tab of the Event Details view):
Jul 31 2014 20:49:04: %PIX-6-302016: Teardown UDP connection 84966377 for outside:18.104.22.168/53 to dmz:10.0.171.165/32793 duration 0:00:01 bytes 281
- Create a new Filter Rule with your expression. In the filter policy, select New/Filter Rule. There are two main parts to the rule. The first group of settings identify what events will match the rule, and the second identifies what to do with events that match our rule. There are two different ways to match an event to our rule.
- Content Strings. Content Strings are simple static string matches. If your unwanted events have an easily-identifiable static string that you can use to identify them, then this is where you should put it. In our example events, the string
%PIX-6-302016: Teardown UDP connection
is seen consistently in all the unwanted events, and we’ll use this as our simple match string.
- PCRE (Perl Compatible Regular Expression): PCRE supports complex expressions that you can use to identify events you'd like to filter. If we configure both a Content String and PCRE, the PCRE is only applied against events that match the Content String first. Content Strings can be used as an effective first-level filter to ensure that the more expensive PCRE operation is only applied to events that pass through the Content String filter. Our string is a simple match, so does not require a regular expression. The simple Content String match will do.
The second part of the Filter rule is defining what to do when the filter rule matches. Available actions include:
- Send Log to Parser. This lets the events matching the expression continue on its usual way to the parser.
- Send Log to ELM. Forward the events matching the expression to the ELM.
- Generate ESM Event. This will directly create an appropriate internal event on the ESM each time this filter rule matches an event. Depending on the type of logs, that can create a very high number of events. Use very carefully.
- Stop Processing Filter Rules. Does not process any more filter rules after this matched.
In our example, we will choose only to send the log to the ELM, and stop processing additional rules for any events that match the content string.
- Enable rules and roll out policy. Finally, ensure that your new filter rules are enabled in the Policy Editor, and then roll out policy by selecting Operations/Rollout. From this point forward, we should see no more of these pesky UDP teardown events.