cancel
Showing results for 
Search instead for 
Did you mean: 
gilbert.gan
Level 7

ESM search and filter is slow

I have ESM deployed in the last 6 months, and when I tried to search my AV logs for a particular infection (e.g. contains(ransom)) by using the filer over a 4 month period, ESM runs for over 20mins and the results are still not displayed.  Is this normal for ESM (or an SIEM) to perform this badly?

0 Kudos
9 Replies
McAfee Employee

Re: ESM search and filter is slow

When doing a 'contains' search you're no longer leveraging the indexing of the ESM which leaves you doing a full table scan. From there it depends upon how much data is being searched and how many resources (CPU/RAM/DiskIO) are available.

Anecdotally, there was resistance for years to allow wildcard searching from the Global Filter for basically this reason (mismatched performance between searches).

0 Kudos
gilbert.gan
Level 7

Re: ESM search and filter is slow

This just seems wrong..

Anyone knows a better way to perform such searches?

0 Kudos
McAfee Employee

Re: ESM search and filter is slow

You can create a dynamic watchlist that performs the search in the background and keeps it updated. Then you can use that watchlist as a filter in views, rules and reports.

0 Kudos
gilbert.gan
Level 7

Re: ESM search and filter is slow

Would this mean the searches apply going forward?  Or i can apply this to historical logs?

0 Kudos
McAfee Employee

Re: ESM search and filter is slow

Here are the steps do set it up. It only takes a minute and I think once you see how it works it will improve your situation.

First, create a new watchlist. I used the term 'TROJAN' for this example. It doesn't need to refresh too often since your Sig-IDs tend to stay fairly static.

watchlist-search1.PNG

Then set the source to ESM Rule Names and enter your string. Regex can be used here.

watchlist-search2.PNG

Go ahead and click Run Now. It usually is pretty quick despite warnings.

watchlist-search3.PNG

Now you can click the Filter icon above the Signature ID field and select your watchlist to filter on.

watchlist-search4.PNG

Results of this example:

watchlist-search5.PNG

It's a couple of extra steps, but it's pretty quick and easy once you know what they are.

0 Kudos
vho
Level 7

Re: ESM search and filter is slow

Hi Andy,

I'm using ESM v9.5.2 and I don't see a Source tab in Watchlist. How do I get this to appear? And did you input AV Signature ID's in the Value tab? Thanks for the suggestion, I've never done it this way.

0 Kudos
McAfee Employee

Re: ESM search and filter is slow

Is it possible that you don't have permissions for the watchlist? Perhaps try as NGCP to verify or you could post a screenshot.

The way this works is that all of the events are queried for the search string. The output of that search is a list of all of the Signature IDs that matched which automatically populate the watchlist when you click Run Now.

Now you can use that watchlist as a filter and since it's an indexed list, it will come back quickly. This is in contrast to doing a brute force table scan against a large volume of events without any indexing efficiencies that the embedded database provides.

0 Kudos
vho
Level 7

Re: ESM search and filter is slow

Hi Andy,

Please see the screenshots with NGCP permissions:

Add Watchlist screenshot.pngAdd Watchlist Value Screenshot.png

0 Kudos
Highlighted
McAfee Employee

Re: ESM search and filter is slow

Thank for for posting the screenshot. Please notice in mine that I have the Dynamic box checked. This should show the tab for you.

0 Kudos