How do you have your data sources designed to accommodate systems with multiple roles? For example, a linux system might host an apache web server and oracle database. That would be three different types of data sources, right?
A consultant told us to create a dummy parent data source for each type of data (i.e. apache, linux, oracle, windows, ....). Then add all the actual system IPs as client data sources under the parent matching the type of data.
Linux Parent with fake IP address
Apache Parent with fake IP address
Oracle Audit Parent with fake IP address
Unfortunately, that doesn't appear to work when the data sources are fed from syslog messages. As the syslog messages come into the receiver, they are matched with the first data source with a matching IP address. For example, the linux data source. The downstream data sources never see the log messages.
I understand why it works that way but now the design our consultant suggested will not work for us.
If messages are forwarded by MEF, can it accommodate multiple client data sources with the same IP address?
The other scenarios that come to mind:
1) Create a standalone data source for each system, and modify the policy so other data source parsers are enabled. For example, create the data source as a linux system and modify the policy to enable the apache and oracle audit parsers. This seems labor and time intensive and you can't tell by looking at the data source what it is actually doing - all you'll see is "linux". This also limits the number of data sources that can be created per receiver compared to the parent/client architecture.
2) Create a parent for each system and create clients for each type of data. For example, for a given system, create a dummy parent and add client data sources for the linux, apache, and oracle data. However, those will all have the same ip address so this option doesn't solve the problem with the syslog stream.
I was really hoping to stick with syslog. All our machines (e.g. linux, oracle, windows with snare) are configured to forward to a central syslog server. That syslog server forwards to our receivers. We would like to keep the central syslog server and syslog forwarding client in play for several reasons:
1) It uses a data storage format we're very familiar with.
2) After ESM points us in the right direction, it is sometimes faster and easier to search the raw logs on the syslog server than try to drill down using ESM or ELM. Particularly when long time periods are involved.
3) It can be expanded and designed with high availability in mind for a relatively small cost.
4) The forwarding client can be used to forward to an ELK instance used for operations monitoring without installing yet another agent.
The original plan was to use Snare, SyslogNG-PE, or NXLOG to forward syslog, windows event log, and ascii log information to the receivers through the syslog server. However, if a syslog stream can't be sent to multiple data sources with the same IP to accommodate the linux-apache-oracle scenario, we'll need to do something else. Either use MEF as a second forwarding agent assuming it can handle the duplicate client IPs or hand modify the policies for each system's data source so they can accommodate all the data types.
Anything I've overlooked?
Thanks for reading.