Showing results for 
Show  only  | Search instead for 
Did you mean: 

The Best way to get bulk historical logs out of Siem

We are doing a firewall remediation project and we need to pull bulk historical logs from Siem for about a 30 day period for specific rules so we can analyze the traffic.  This is millions and millions of lines.  Pulling this data from the front end has been difficult due to the data size limitations of Siem and is putting huge stress on our system.  Does anyone have any experience with trying to pull large data sets from the tool?  Advice? We are also exploring how this data could be pulled from the DB.  Has anyone done this and can advise?  I know the log data is stored in a proprietary format so is there a conversion tool out there that can be used to convert the data to a standard format like csv or excel?

2 Replies
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 2 of 3

Re: The Best way to get bulk historical logs out of Siem

I have not tested in pre-11.x versions, but the API can be utilized to query the data

qryExecuteDetail (gives you the result id needed for retrieving status and results, here you define search parameters, and tell it the fields to return)
qryGetStatus (give it the result id and it returns the status of the query)
qryGetResults (give it the result id once the status returns complete, define the startPos and numRows to return)
qryClose (close the query after retrieving the results or you will have multiple queries cached impacting the system performance)

Hope this helps point you in the right direction on how to programmatically return results with massive data sets.

If you are curious on how to get the data needed on the fields, the following APIs will get you that information

You can access the help contents by going to https://youresm/rs/esm/v2/help

Don't forget to authenticate first to https://youresm/rs/esm/login

More documentation can be found here.

Reliable Contributor
Reliable Contributor
Report Inappropriate Content
Message 3 of 3

Re: The Best way to get bulk historical logs out of Siem

Do you want the aggregated normalized events or the raw events from the ELM?

You Deserve an Award
Don't forget, when your helpful posts earn a kudos or get accepted as a solution you can unlock perks and badges. Those aren't the only badges, either. How many can you collect? Click here to learn more.

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community