I have ESM deployed in the last 6 months, and when I tried to search my AV logs for a particular infection (e.g. contains(ransom)) by using the filer over a 4 month period, ESM runs for over 20mins and the results are still not displayed. Is this normal for ESM (or an SIEM) to perform this badly?
When doing a 'contains' search you're no longer leveraging the indexing of the ESM which leaves you doing a full table scan. From there it depends upon how much data is being searched and how many resources (CPU/RAM/DiskIO) are available.
Anecdotally, there was resistance for years to allow wildcard searching from the Global Filter for basically this reason (mismatched performance between searches).
You can create a dynamic watchlist that performs the search in the background and keeps it updated. Then you can use that watchlist as a filter in views, rules and reports.
Here are the steps do set it up. It only takes a minute and I think once you see how it works it will improve your situation.
First, create a new watchlist. I used the term 'TROJAN' for this example. It doesn't need to refresh too often since your Sig-IDs tend to stay fairly static.
Then set the source to ESM Rule Names and enter your string. Regex can be used here.
Go ahead and click Run Now. It usually is pretty quick despite warnings.
Now you can click the Filter icon above the Signature ID field and select your watchlist to filter on.
Results of this example:
It's a couple of extra steps, but it's pretty quick and easy once you know what they are.
I'm using ESM v9.5.2 and I don't see a Source tab in Watchlist. How do I get this to appear? And did you input AV Signature ID's in the Value tab? Thanks for the suggestion, I've never done it this way.
Is it possible that you don't have permissions for the watchlist? Perhaps try as NGCP to verify or you could post a screenshot.
The way this works is that all of the events are queried for the search string. The output of that search is a list of all of the Signature IDs that matched which automatically populate the watchlist when you click Run Now.
Now you can use that watchlist as a filter and since it's an indexed list, it will come back quickly. This is in contrast to doing a brute force table scan against a large volume of events without any indexing efficiencies that the embedded database provides.