cancel
Showing results for 
Search instead for 
Did you mean: 

Deleting a data source vs record count in ESM database

Hello,

I could not find this information in the documentation, but when I asked a McAfee SIEM expert, I was told that deleting a data source in a receiver would automatically delete that source's data in the ESM database.

To save space in the ESM database, I decided to delete one extremely verbose data source.

The source was deleted and doesn't appear in the device tree anymore, but it's data does not seem to have been deleted from the ESM database.

In a dashboard showing "event count by device" with a timeframe of "All", I had the following numbers for this source just before I deleted it :

     Count: 1 123 394 661;     Total Event Count: 4 362 312 642

My understanding is that the ESM had 1.12 billion records representing 4.36 events in its database for this source.

I expected the number of event records in the database (shown in System properties -> Database -> Memory Use -> Events Total Records) to decrease by 1.12 billion but the number of records is increasing instead of decreasing. I understand that new records are being inserted in the database but I am surprised that the total number of records is not decreasing while the deleted source's events are (presumably)  being deleted.

Is the deletion of these records taking place in the background as a low priority process and I have to be patient to see the results in the "Events Total Records" field of the Database Memory Use ?

In case it helps the event collection rate for all the sources on our two receivers for the last 30 minutes is around 8000 events per second. That's events, not records - since our average aggregation rate is 5 to 1, it's approximately 1600 records / sec or around 1 million records per 10-min interval. The total number of event records is 3.9 billion, so it's impossible that I don't see the effect of deleting 1.1 billion records which represent more that ont third of the records in the database.

Thank you very much for any hep you can provide.

Marc Mazuhelli

Université de Sherbrooke

1 Reply
rth67
Level 12
Report Inappropriate Content
Message 2 of 2

Re: Deleting a data source vs record count in ESM database

It was my understanding that if you deleted a Data Source that still had data associated with it in the database, it would simply orphan that data.

When we decommission a system, we simply change the IP address on it, disable it from collecting, and rename the "Name" field adding "zz-" at the front, and a date code at the end to denote when we disabled collection.

This allows us to still report on the historical data until it is purged from the database (following our overall purge parameters).