Our efforts up until now have focused on optimizing your environment with a small initial set of data sources coming into the system. By performing initial configuration and tuning in the early stages under light load, you gain the benefits of a more nimble environment, and reduced impact if you make a mistake.
Once you have built up a usable environment with your initial sets of data, you are ready to begin adding additional data sources. As you ramp up your data sources, it’s wise to prioritize data sources based on your desired use cases, ensuring that the data sources that will deliver the most value are introduced early.
Bring new data sources online at a measured pace. This helps ensure you don’t overwhelm your environment with unexpected volumes, and allows you to make adjustments to your strategy as your ramp things up.
By now you are familiar with adding data sources to ESM manually. As you add new data sources, manual methods may become unworkable. ESM provides several avenues for importing data sources in bulk.
Bulk Data Source Import via CSV
When you have large volumes of data source, it is often convenient to bulk import. The SIEM Receiver supports importing data source definitions via a CSV-formatted file.
- If you have not already done so, manually define a single representative example of each data source you would like to bulk import.
- Export your Receiver data source definitions to a CSV (Receiver Properties/Data Sources/Export).
- Open the CSV in a text editor to review the format for the import file. At the top of the file you’ll find a header line that defines the fields for you.
- Edit the file and add a line for each data source you would like to import, using an existing pre-defined data source as a template. Ensure that the first field in each line (“op”) reads “add”. This field is the operation you are asking the Receiver to perform.
Note: It is NOT recommended to use Microsoft Excel to edit the CSV file. The rec_id column (which uniquely identifies your Receiver) is a 19-digit number. Excel interprets this as an integer, and renders it in scientific notation when the CSV is saved. For best results, use a text editor or other tool to modify the CSV.
- Save the file and import it into the Receiver (Receiver Properties/Data Sources/Import). You will be prompted to write configuration to your Receiver, and roll out policy.
Auto Learn Data Sources
By default, McAfee SIEM receivers block packets from IP addresses that don’t have an associated data source configured in the SIEM. This provides a measure of DoS protection, and ensures that well-meaning data source owners can’t overwhelm your SIEM with data that you aren’t prepared to bring in.
SIEM Receivers support Auto Learn of data sources. When enabled, the Receiver listens for event traffic from any source. Any devices that are sending logs that are not associated with an existing configured data source are cataloged. Auto-learned data sources may be reviewed and added to the Receiver manually by SIEM administrators, or added automatically via defined sets of rules. Below is a brief video describing the use of Auto Learn.
Enable Auto Learn
To enable Auto Learn, open Receiver properties, select Data Sources/Auto Learn. You will be presented with the Auto Learn UI, which will initially be empty. To turn on Auto Learn, click the Enable button next to the protocols of your choice.
Once Auto Learn is enabled, new data sources will begin to appear in the UI as they are discovered. In the example below, the Receiver has seen logs from 2 Cisco PIX firewalls, and one Linux device.
Manually add Auto Learned data sources
Once data sources are discovered, it is a simple process to add the new data source definition. Select the discovered source (multi select via shift-click and control-click is supported), and click Add.
The first dialog you will see prompts you to select how the new data source should be created: either as a new parent data source, or as a client of an existing data source. For now, select “Create data sources for the selected auto learned items”.
Note: in some environments with large numbers of data sources, or high event rates, it may be advantageous to take advantage of client data sources. Data source hierarchy is outside the scope of this document.
Next you will be presented with an abridged version of the standard Add Data Source wizard you would see if you were adding a data source in the standard manual method. Make any changes necessary, and click OK. Your newly learned data source definitions will be added to the Receiver and event collection will begin.
Automatically add Auto Learned data sources
In circumstances where you have large numbers of similar data sources, you may find it useful to automatically add data sources to the Receiver as they begin sending logs. To begin, open the Auto Learn UI (Receiver Properties/Data Sources/Auto Learn), and click “Configure rules for auto adding data sources”.
Auto Add Rules allow you to define parameters for the data sources that the Receiver will be creating. There are two main parts to an Auto Add Rule:
- Matching criteria. This defines what data sources the rule will apply to. In our example above, we see that the Auto Learn engine is identifying several Cisco PIX devices that are sending logs to the SIEM. We’ll set up a rule to automatically add Cisco PIX devices in the future.
- Data Source Creation Parameters. Once an incoming learned data source has been matched to a rule via Matching Criteria, we need to provide some parameters that the Receiver can use to flesh out the unique configuration for each incoming data source. In our example we’ll define a name convention for our new data sources that includes the IP address, as shown above.
With our new rule, as new PIX devices begin to send logs to the SIEM, they will be automatically matched to our rule, and data sources will be created as shown below, without administrator interaction.