4 Replies Latest reply on Oct 14, 2016 8:37 AM by itzamlan

    How data flow to correlation engine

      Dear guys,

       

      I have a few questions regarding how data flow to correlation engine. From my little understanding, here's a list of steps I think things should be:

       

      From Event Receiver to Correlation Engine (For both Correlation Engine on ERC and ACE in case of real-time rule-based correlation):

      1. ERC gets or receives events from various data sources. ERC do their parsing job.
      2. ESM pulls parsed events from ERC in a timely manner.
      3. As soon as ESM got events, it will send all events to correlation engine it knew e.g. both on ERC and ACE if there are correlation engine for both. (I'm not sure about this step. Will ESM push events to correlation engine or correlation engine will pull from ESM?)
      4. ESM pulls correlation events from correlation engine if any rule matched and correlation events produced.

       

       

      In my opinion, correlation engine, likes any other data source object, got events and processing according to rules. However, in case of correlation engine it isn't rule for parsing but rule for correlation.

       

       

      Here are my questions:

      1. Could someone confirm my data flow above? especially step No. 3. I want to know if it's an ESM who push event to correlation engine or correlation engine who pull events from ESM?
      2. In case of local correlation engine on ERC, since we can think of correlation engine as another ordinary data source; correlation engine data source will consume EPS on ERC likes any other data source, correct?
      3. If it's true for question No. 2, it's mean if we going to use correlation engine on ERC, do we have to sizing double EPS on ERC? (All events from data sources + parsed events from ESM so it's double).
      4. If my assumption on question No. 2 isn't absolutely correct, how much performance impact on ERC if we going to use correlation engine data source? If customer can't afford ACE in the first place how can we sizing ERC in this case?
      5. If there are multiple ERC, can correlation engine on a single ERC perform correlation for all events coming from all ERC? Since all events will be pass through single ESM? e.g. we can place a bigger model of ERC for HQ and small ones for branches; then we let alone bigger ERC on HQ to do correlation for all events.

       

      Thank you in advanced for answering my questions.

       

      Best regards,

       

      Parinya

        • 1. Re: How data flow to correlation engine
          Chris Boldiston

          Hi Parinya

           

           

          I will get back to you with the answers to those questions.

           

           


          Thanks

           

           

          Chris

          • 2. Re: How data flow to correlation engine
            Chris Boldiston

            Hi Parinya

             

             

            I have posted some answers / information inline for your questions. My strong recommendation is to talk with a McAfee Enterprise Solutions Architect as they have the expert knowledge and experience gained from designing and implementing SIEM solutions at our customer sites. Please let me know if you would like me to get them in contact with you.

             

            From Event Receiver to Correlation Engine (For both Correlation Engine on ERC and ACE in case of real-time rule-based correlation):

             

            1 ERC gets or receives events from various data sources. ERC do their parsing job.
            2 ESM pulls parsed events from ERC in a timely manner.
            3 As soon as ESM got events, it will send all events to correlation engine it knew e.g. both on ERC and ACE if there are correlation engine for both. (I'm not sure about this step. Will ESM push events to correlation engine or correlation engine will pull from ESM?)
            >> The ESM will feed events to the ACE. If you are using correlation on a receiver there are 2 ways that it can get events. If you have the option selected to "Use Local Data" then events go to the correlation engine after they have been parsed by the receiver. However, if you do not have that selected then the ESM will feed events to the correlation engine in the same way as it feeds ACE.
            4 ESM pulls correlation events from correlation engine if any rule matched and correlation events produced.

            In my opinion, correlation engine, likes any other data source object, got events and processing according to rules. However, in case of correlation engine it isn't rule for parsing but rule for correlation.

            Here are my questions:

            1 Could someone confirm my data flow above? especially step No. 3. I want to know if it's an ESM who push event to correlation engine or correlation engine who pull events from ESM?
            >> See point 3 above.

            2 In case of local correlation engine on ERC, since we can think of correlation engine as another ordinary data source; correlation engine data source will consume EPS on ERC likes any other data source, correct?
            >> That is correct but the correlation engine is getting events from all datasources so it consumes more resources than just one data source.  We recommend the stand alone ACE to for optimum system performance.

            3 If it's true for question No. 2, it's mean if we going to use correlation engine on ERC, do we have to sizing double EPS on ERC? (All events from data sources + parsed events from ESM so it's double).
            >> Doubling the sizing is typically adequate if you are using a receiver correlation engine.

            4 If my assumption on question No. 2 isn't absolutely correct, how much performance impact on ERC if we going to use correlation engine data source? If customer can't afford ACE in the first place how can we sizing ERC in this case?
            >> That would depend on a number of factors which could vary at customer sites such as the number of rules enabled and the custom rules which have been created.  A good rule is the total ESM EPS is what will need to be allocated to the REC Correlation engine.  The ACE will be the most efficient and scalable tool to handle this task.

            5 If there are multiple ERC, can correlation engine on a single ERC perform correlation for all events coming from all ERC? Since all events will be pass through single ESM? e.g. we can place a bigger model of ERC for HQ and small ones for branches; then we let alone bigger ERC on HQ to do correlation for all events.
            >> Yes, a single ERC can correlate on events from all REC. Putting the bigger REC at HQ is one approach especially if this will be a dedicated correlation receiver. However, if it’s going to be primarily for correlation then an ACE is recommended as that is the most efficient device for this function. The ACE not only offers correlation rule capability but also the ability to monitor risk in your environment. The ACE can also correlate past events with its historical correlation function.

             

             

             

            Regards

             

             

            Chris

            • 3. Re: How data flow to correlation engine

              Thank you very much!!

              • 4. Re: How data flow to correlation engine
                itzamlan

                Hi Chris,

                 

                I have no exposure to deviation and historical correlation rules.

                 

                1) How does the deviation component of the ACE work?

                 

                2) And do we need to feed some sort of profile to check the deviation? Need an example.

                 

                3) And also for the risk-based(Event-based) correlation and historical correlation does the ACE has some kind of buffer to store risk comparisions based upon some parameters or it periodically pulls events from ELM via ESM for the same? Does it impose any sort of overhead on any of the other appliance?

                 

                4) And how the historical correlation works? Is the historical correlation is kinda versioning older & newer versions of correlation rules?

                 

                Thanks in advance.