1 Reply Latest reply on Apr 6, 2017 3:53 AM by abanaru

    Delayed or Historical Correlation




      I have an interesting scenario that I am not certain of the answer, and was hoping the community might be able to give some advice on. 


      Problem:  How to go about correlating multiple events from different log sources when one log source is delayed by 24 hours (e.g. batch upload of 24 hours of logs)?


      Option 1: Delayed Correlation

      I have previously looked up the time-sensitivity of correlation rules and came across this resource for best practices of an ACE.  Within the document, the definition for "Time Order Tolerance" was given to account for events that arrive out of sequence (or late) within a certain time threshold.  The default is set to 60 minutes, but less is preferred since it is it has an expensive cost in terms of memory usage.  It is because of this that I believe that this would be impractical to meet a 24 hour delay.


      Option 2: Historical Correlation

      Another document I found identified the ability to enable historical correlation.  The ACE (Advanced Correlation Engine) has the ability to perform historical correlation, and can execute the current correlation rule set against a configurable time frame of data.  There is a price to pay with this option though:

      • Enabling historical correlation switches the mode from real-time to historical until the job is finished
      • There is no catch up period.  Logs received during the period where the historical correlation is executing will not pass through the correlation engine (unless another historical scan is executed).


      It is possible to run both a real-time correlation and historical correlation engine, but it requires two ACE devices to do so simultaneously.  Unfortunately, my organization only runs with one ACE and there is likely no appetite to switch modes unless absolutely necessary. 



      Has anyone else encountered a similar problem?  Are there any other methods that I have not thought of here?


      Thank you.

        • 1. Re: Delayed or Historical Correlation

          I can't think of any other way than Option 1 which is not so feasible because of the big time-window.

          Historical correlation won't help you because it works the same(on correlation rules) as real time but for already stored events.


          Try Option 1 and tell us how it goes.


          As a side note, you could add a local correlation engine to your receiver and then switch the ACE to historical if you don't have any risk based correlation in place (because the local correlation engine can't handle risk correlation).