1 Reply Latest reply on Dec 11, 2013 3:03 PM by rth67

    Deleting a data source vs record count in ESM database

    marcmazu

      Hello,

       

      I could not find this information in the documentation, but when I asked a McAfee SIEM expert, I was told that deleting a data source in a receiver would automatically delete that source's data in the ESM database.

       

      To save space in the ESM database, I decided to delete one extremely verbose data source.

      The source was deleted and doesn't appear in the device tree anymore, but it's data does not seem to have been deleted from the ESM database.

      In a dashboard showing "event count by device" with a timeframe of "All", I had the following numbers for this source just before I deleted it :

           Count: 1 123 394 661;     Total Event Count: 4 362 312 642

       

      My understanding is that the ESM had 1.12 billion records representing 4.36 events in its database for this source.

       

      I expected the number of event records in the database (shown in System properties -> Database -> Memory Use -> Events Total Records) to decrease by 1.12 billion but the number of records is increasing instead of decreasing. I understand that new records are being inserted in the database but I am surprised that the total number of records is not decreasing while the deleted source's events are (presumably)  being deleted.

       

      Is the deletion of these records taking place in the background as a low priority process and I have to be patient to see the results in the "Events Total Records" field of the Database Memory Use ?

       

      In case it helps the event collection rate for all the sources on our two receivers for the last 30 minutes is around 8000 events per second. That's events, not records - since our average aggregation rate is 5 to 1, it's approximately 1600 records / sec or around 1 million records per 10-min interval. The total number of event records is 3.9 billion, so it's impossible that I don't see the effect of deleting 1.1 billion records which represent more that ont third of the records in the database.

       

      Thank you very much for any hep you can provide.

       

      Marc Mazuhelli

      Université de Sherbrooke

        • 1. Re: Deleting a data source vs record count in ESM database
          rth67

          It was my understanding that if you deleted a Data Source that still had data associated with it in the database, it would simply orphan that data.

          When we decommission a system, we simply change the IP address on it, disable it from collecting, and rename the "Name" field adding "zz-" at the front, and a date code at the end to denote when we disabled collection.

          This allows us to still report on the historical data until it is purged from the database (following our overall purge parameters).