9 Replies Latest reply on May 19, 2016 2:53 PM by andy777

    ESM search and filter is slow

    gilbert.gan

      I have ESM deployed in the last 6 months, and when I tried to search my AV logs for a particular infection (e.g. contains(ransom)) by using the filer over a 4 month period, ESM runs for over 20mins and the results are still not displayed.  Is this normal for ESM (or an SIEM) to perform this badly?

        • 1. Re: ESM search and filter is slow
          andy777

          When doing a 'contains' search you're no longer leveraging the indexing of the ESM which leaves you doing a full table scan. From there it depends upon how much data is being searched and how many resources (CPU/RAM/DiskIO) are available.

           

          Anecdotally, there was resistance for years to allow wildcard searching from the Global Filter for basically this reason (mismatched performance between searches).

          • 2. Re: ESM search and filter is slow
            gilbert.gan

            This just seems wrong..

             

            Anyone knows a better way to perform such searches?

            • 3. Re: ESM search and filter is slow
              andy777

              You can create a dynamic watchlist that performs the search in the background and keeps it updated. Then you can use that watchlist as a filter in views, rules and reports.

              • 4. Re: ESM search and filter is slow
                gilbert.gan

                Would this mean the searches apply going forward?  Or i can apply this to historical logs?

                • 5. Re: ESM search and filter is slow
                  andy777

                  Here are the steps do set it up. It only takes a minute and I think once you see how it works it will improve your situation.

                   

                  First, create a new watchlist. I used the term 'TROJAN' for this example. It doesn't need to refresh too often since your Sig-IDs tend to stay fairly static.

                  watchlist-search1.PNG

                   

                  Then set the source to ESM Rule Names and enter your string. Regex can be used here.

                  watchlist-search2.PNG

                   

                  Go ahead and click Run Now. It usually is pretty quick despite warnings.

                  watchlist-search3.PNG

                  Now you can click the Filter icon above the Signature ID field and select your watchlist to filter on.

                  watchlist-search4.PNG

                   

                  Results of this example:

                  watchlist-search5.PNG

                   

                  It's a couple of extra steps, but it's pretty quick and easy once you know what they are.

                  • 6. Re: ESM search and filter is slow
                    vho

                    Hi Andy,

                     

                    I'm using ESM v9.5.2 and I don't see a Source tab in Watchlist. How do I get this to appear? And did you input AV Signature ID's in the Value tab? Thanks for the suggestion, I've never done it this way.

                    • 7. Re: ESM search and filter is slow
                      andy777

                      Is it possible that you don't have permissions for the watchlist? Perhaps try as NGCP to verify or you could post a screenshot.

                       

                      The way this works is that all of the events are queried for the search string. The output of that search is a list of all of the Signature IDs that matched which automatically populate the watchlist when you click Run Now.

                       

                      Now you can use that watchlist as a filter and since it's an indexed list, it will come back quickly. This is in contrast to doing a brute force table scan against a large volume of events without any indexing efficiencies that the embedded database provides.

                      • 8. Re: ESM search and filter is slow
                        vho

                        Hi Andy,

                         

                        Please see the screenshots with NGCP permissions:

                        Add Watchlist screenshot.pngAdd Watchlist Value Screenshot.png

                        • 9. Re: ESM search and filter is slow
                          andy777

                          Thank for for posting the screenshot. Please notice in mine that I have the Dynamic box checked. This should show the tab for you.