Taking the old rows from db table

[gilevi]

Hello,

I'm trying to take the old rows from db table.

for now, the collector just taking the new rows and send it to the siem but i need the olds too, at least one time.

the exist xml file (the siemcollector config file) query is : 

<Query>SELECT [dbo].[table].[action],       from table  etc...

I want to change it to :

<Query>SELECT top (5000) [dbo].[table].[action],  from table

when Im trying to change it manually it is getting overide by the siemcollector settings again and again.

any ideas?

thanks

[lratcliffe · Accepted solution]

Change the bookmark instead of the query.  The bookmark file is in <siemcollectorinstalldirectory>/plugins/<pluginid>/

It's a plain text file and contains a value which matches your selected bookmark field.  Stop the SIEM Collector service, edit this file and start the service again.

You will receive duplicate events - i.e. anything that you have already collected will be collected a second time as SIEM Collector reads forwards through your database.  The volume of data retrieved could cause performance issues on the database, the system running SIEM Collector, your Receiver, ACE, ELM and ESM.  Due to this try to limit the retrieval of old data to the minimum amount possible.

Also check for the setting that Restricts insertion of Historical Data - on SIEM 10.x this is in the ESM properties under the Database tab and on 11.x it is in the Receiver properties on the Events, Flows and Logs tab.  If historical insertion is restricted, it is likely that this older data will never make it to your ESM.

[gilevi]

Thanks!!

[itzikn]

i'm trying to collect rows from large db table but see the bookmark file value forwarding by 5000 once in 5 minutes, the rows on this table fills up faster then the collector collect and the siem is not at a pace can i do somthing to change it for collecting more then 5000 in 5 minutes?