The Best way to get bulk historical logs out of Siem
We are doing a firewall remediation project and we need to pull bulk historical logs from Siem for about a 30 day period for specific rules so we can analyze the traffic. This is millions and millions of lines. Pulling this data from the front end has been difficult due to the data size limitations of Siem and is putting huge stress on our system. Does anyone have any experience with trying to pull large data sets from the tool? Advice? We are also exploring how this data could be pulled from the DB. Has anyone done this and can advise? I know the log data is stored in a proprietary format so is there a conversion tool out there that can be used to convert the data to a standard format like csv or excel?
Re: The Best way to get bulk historical logs out of Siem
I have not tested in pre-11.x versions, but the API can be utilized to query the data
qryExecuteDetail (gives you the result id needed for retrieving status and results, here you define search parameters, and tell it the fields to return) qryGetStatus (give it the result id and it returns the status of the query) qryGetResults (give it the result id once the status returns complete, define the startPos and numRows to return) qryClose (close the query after retrieving the results or you will have multiple queries cached impacting the system performance)
Hope this helps point you in the right direction on how to programmatically return results with massive data sets.
If you are curious on how to get the data needed on the fields, the following APIs will get you that information qryGetSelectFields qryGetFilterFields
You can access the help contents by going to https://youresm/rs/esm/v2/help
Don't forget to authenticate first to https://youresm/rs/esm/login