Showing results for 
Search instead for 
Did you mean: 

The Best way to get bulk historical logs out of Siem

We are doing a firewall remediation project and we need to pull bulk historical logs from Siem for about a 30 day period for specific rules so we can analyze the traffic.  This is millions and millions of lines.  Pulling this data from the front end has been difficult due to the data size limitations of Siem and is putting huge stress on our system.  Does anyone have any experience with trying to pull large data sets from the tool?  Advice? We are also exploring how this data could be pulled from the DB.  Has anyone done this and can advise?  I know the log data is stored in a proprietary format so is there a conversion tool out there that can be used to convert the data to a standard format like csv or excel?

2 Replies
McAfee Employee TaskManager
McAfee Employee
Report Inappropriate Content
Message 2 of 3

Re: The Best way to get bulk historical logs out of Siem

I have not tested in pre-11.x versions, but the API can be utilized to query the data

qryExecuteDetail (gives you the result id needed for retrieving status and results, here you define search parameters, and tell it the fields to return)
qryGetStatus (give it the result id and it returns the status of the query)
qryGetResults (give it the result id once the status returns complete, define the startPos and numRows to return)
qryClose (close the query after retrieving the results or you will have multiple queries cached impacting the system performance)

Hope this helps point you in the right direction on how to programmatically return results with massive data sets.

If you are curious on how to get the data needed on the fields, the following APIs will get you that information

You can access the help contents by going to https://youresm/rs/esm/v2/help

Don't forget to authenticate first to https://youresm/rs/esm/login

More documentation can be found here.

Reliable Contributor brenta
Reliable Contributor
Report Inappropriate Content
Message 3 of 3

Re: The Best way to get bulk historical logs out of Siem

Do you want the aggregated normalized events or the raw events from the ELM?

More McAfee Tools to Help You
  • Subscription Service Notification (SNS)
  • How-to: Endpoint Removal Tool
  • Support: Endpoint Security
  • eSupport: Policy Orchestrator
  • Community Help Hub

      New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

    • Find Forum FAQs
    • Learn How to Earn Badges
    • Ask for Help
    Go to Community Help

    Join the Community

      Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

    • Get helpful solutions from McAfee experts.
    • Stay connected to product conversations that matter to you.
    • Participate in product groups led by McAfee employees.
    Join the Community
    Join the Community