How buffer in event receiver architecturing

[parinya.ekparin]

I'm not sure about how our receiver architecturing. Is there any buffer before event will be picked up for processing.

I want to know more about its buffer architecture to answer following questions:

• How much percentage of EPS burst can receiver handle and also how long can it take?
  For example, if box can handle  20% EPS burst, ERC1250 might take 6000 EPS during burst.
  Please note that I'll sizing based on EPS on the data sheet. I know that this is a extra EPS solely for burst only and should not take into account when we sizing hardware.
• If the connection between ESM and Receiver is loss, will events cached on event receiver?
• If receiver caches events, what's used for caching, Memory or disk?
• How long can receiver cache events? Any example case?

Thank you in advacned.

Regards,

Parinya

Message was edited by: parinya.ekparinya on 11/22/12 4:53:04 AM CST

[mmalan · Accepted solution]

Hello Parinya,

I'm not sure if you are looking for technical specs of a particular receiver model or just looking for general "How Does it Work?" type information.

For the majority of the Receiver data sources (i.e. syslog,Microsoft wmi, McAfee ePO etc..), the event data is first written or "cached" to a file on disk. The data is then parsed and inserted into the Receiver Database. Next the data is transferred from the Receiver Database to the ESM Database and finally displayed in the User Interface.

Really, the only limiting factors to the amount of data a Receiver can "cache" during traffic bursts or otherwise are the NIC capabilities, the Receiver Disk Capacity and the Database Record Capacity.

*The NIC is limited to 1GB network traffic speeds.

*The disk capacity is limited to its data RAID size(typically between 500GB and 1TB).

*The Database is limited to 250 million records.

The EPS ratings are an approximate limitation to the rate at which data can be inserted into database.

So to answer your questions specifically...

* How much percentage of EPS burst can receiver handle and also how long can it take?

- A "burst" can last as long as it does not cause the device to exceed the above mentioned limitations.

* If the connection between ESM and Receiver is loss, will events cached on event receiver?

- If the connection between the ESM and Receiver is lost, the Receiver Database will "cache" the event data until 250 Million records have been reached from the last successful data transfer.

* If receiver caches events, what's used for caching, Memory or disk?

- The receiver will first cache the incoming event data in file format on disk and then once it is written to the database the files are removed. The events are then stored in the receiver database until the record count has reached the 250 million count at which point the oldest data is overwritten with the newest.

*How long can receiver cache events? Any example case?

- The receiver can cache events as long as the maximum record count is not exceeded. I have seen receivers with 2 to 3 years of data on them and others that roll the data on a monthly basis. It just depends on how much data is coming in and how fast the 250 million record count is reached.

Let me know if this is what you were looking for.

Thanks,

Matt

[parinya.ekparin]

Actually, what I'm looking for is "How Does it Work?".

What you told me is exactly what I want to know.

Thank you very much for useful information.

Best regards,

Parinya

[parinya.ekparin]

Hi Matt,

If you don't mide, I have a few more questions:

• For 250 million records in receiver database, does a model matter? For example, a bigger model got a bigger number, said 300 million of records; or we can use the same number for every model? At least, I know that a bigger model have higher event processing rate.
• As far as I know, ESM will pull events from receiver in a timely manner. According to information you provided, it's mean ESM will connect and get events data from receiver database. Am I correct? So if a receiver cannot process events in time (I mean events still reside in a data file on receiver, thus they do not get parsed and picked up to database yet.), those events will have to wait for a next round of ESM pulling? Said if ESM was configured to pull events every 10 minutes (by default), those events have to wait for another 10 minutes before ESM will pull them from receiver database? This question is a complicate one. I apologize if I cannot explain it clearly enough.
• How about raw events from receiver to ELM? Will receiver initiate connection and send raw events to ELM; or ELM will be the one who initiate and pull from receiver? As far as I know receiver have to wait for total size of events per data source comes above 5MB before sending it to ELM, otherwise they have to wait for 12 hours before send raw events to ELM. And because of they're raw events, is it data file they send to ELM? So in this case 250 million records doesn't matter. A bigger model of receiver got a bigger disk so will its cache last longer for raw events

Thank you in advacned.

Best regards,

Parinya

[spetting]

Parinya,

Below are the answers to your questions.

• For 250 million records in receiver database, does a model matter? For example, a bigger model got a bigger number, said 300 million of records; or we can use the same number for every model? At least, I know that a bigger model have higher event processing rate.
  • The 250 Million records is an average number for software running 9.0.0 or higher. The actual number of records will vary depending on the hardware model. I do not have a list per model, but the largest Receivers will hold 500 Million Records and the smallest Receivers will hold 100 Million Records.
• As far as I know, ESM will pull events from receiver in a timely manner. According to information you provided, it's mean ESM will connect and get events data from receiver database. Am I correct? So if a receiver cannot process events in time (I mean events still reside in a data file on receiver, thus they do not get parsed and picked up to database yet.), those events will have to wait for a next round of ESM pulling? Said if ESM was configured to pull events every 10 minutes (by default), those events have to wait for another 10 minutes before ESM will pull them from receiver database? This question is a complicate one. I apologize if I cannot explain it clearly enough.
  • You are correct that the ESM will connect to the receiver and pull events from the receiver database. If the receiver has not parsed the data by the time the ESM collects it, the data will be processed and inserted in the database and the ESM will grab it the next time it collects the data.
• How about raw events from receiver to ELM? Will receiver initiate connection and send raw events to ELM; or ELM will be the one who initiate and pull from receiver? As far as I know receiver have to wait for total size of events per data source comes above 5MB before sending it to ELM, otherwise they have to wait for 12 hours before send raw events to ELM. And because of they're raw events, is it data file they send to ELM? So in this case 250 million records doesn't matter. A bigger model of receiver got a bigger disk so will its cache last longer for raw events
  • The ELM raw logs are processed a little bit different than how the events are processed. The logs are collected and inserted into a file on disk. The data is then parsed and events are inserted into the receiver database. This same parsing process that inserts the events in the database also checks to see if the raw log needs to be sent to the ELM. If it does, it will write that raw log to another file on disk where it will wait to be sent to the ELM. The process of getting the logs to the ELM is initiated by the ELM. The ELM checks every few minutes to see if the receiver has any files ready to be sent to the ELM. If there are any that have reached 5MB in size or are 12 hours old, they will be sent to the ELM.
  • The database holds a set number of event records. ELM Raw logs do not have a set limit. The only limit on the number of raw ELM logs the Receiver can store, in the case of communication loss to the ELM, is the size of the receiver disk.

Let me know if you have any additions questions.

Thanks,

Steve