4 Replies Latest reply on Mar 17, 2014 11:22 AM by dmease729

    Data enrichment clarification - where the enrichment field is added

    dmease729

      Hi,

       

      I have recently been on the training course for ESM and am looking to play around with this feature in order to get more familiar.  In my ESM build (9.3.0) I have ePO added as a device, and am focusing on VirusScan Enterprise.  When looking at the events within ESM, my current scenario is that I want to quickly see what VSE DAT level the host has that generated the original alerts.  Using default event aggregation settings (SigID, Source IP, Destination IP), the hostname should be present in the aggregated events as the Source IP in question is the source IP of the host detecting the malware.  I also have two test endpoints, both of which I have tested with EICAR.

       

      Questions at the bottom of this post, with screenshots of my current configuration below:

       

      For the data enrichment, I have configured as follows:

       

      ESM > System Properties > Data Enrichment > Add

       

      01 - Main.jpgNote: 15 minutes is set in a lab environment - I need to review impact/benefit of having a relatively short trigger rate.

      02 - Source.jpg

       

      03 - query.jpgNote: query may need to be tidied up - it is a cut down view of a default SQL view within the ePO DB.

      04 - query test.jpgConfirming connection and DB query results.

      05 - destination.jpgNote: not sure here if this should be the ERC that is the selected device.  The Product Guide p59 simply states "select the devices you want to write the data enrichment rules to".  VSE DAT is a custom field I have added, discussed further below.

       

      06 - devices and rule.jpgNote: same as above - only the ePO server 'device' selected for now.

       

       

      ESM > System Properties > Custom types

       

      Existing types 'Analyzer_DAT_Version' and 'DAT_Version' are both decimal data types, and the DAT field within the ePO DB is a string.  Therefore, I created a custom type as below.  This may need to be reviewed as as per the warning, 'Random' string may need to be used - although the string will not change very frequently, it will change a lot over time.  Not sure about this one at the moment.

       

      07 - custom type.jpg

       

       

       

      Questions:

       

      - As per the 9.3.0 Product Guide p58, data acquisition takes place on the ESM, and not on the devices.  When the data acquisition takes place, are the stored events within ESM themselves enriched, and if so are all of them enriched or only events from a specific time period relevant to the acquisition, or is it the event displays that are enriched (ie the events themselves are left untouched, but when we view the events in a drill down view, the relevant custom fields are added)

      - If I set the device to enrich to be the ePO server, will the events I am looking to enrich actually be enriched, as within the ESM events themselves, the actual device is '<EPO server> - <EPO server>_VirusScan (ePO)'.  Ie - which device should I select for this specific case? If I select the event receiver that is sending the aggregated events to ESM, will that work or not (I am assuming not).

      - Just for completeness, as the enrichment process is actually taking place on the ESM, I am assuming that if a specific field is missing (or may go missing) as a result of aggregation, then using it as a lookup field will not work.  This makes sense to me, but there may be things i am not taking into account - I am still fairly new to this!

       

       

      Thanks in advance!

        • 1. Re: Data enrichment clarification - where the enrichment field is added
          dmease729

          To highlight my main source of confusion - the documentation points to everything happening on the ESM, yet when I configure a data enrichment rule, then select Write all Data Enrichment settings to devices, the only device I can apply the data enrichment rule to is my receiver.  If everything happens on the ESM, why are the rules written to the receiver??

          • 2. Re: Data enrichment clarification - where the enrichment field is added
            dmease729

            Update: For some reason I hadnt seen this before, but the custom types tab within the ESM events for the VSE 'device' actually shows the Analyzer_DAT_Version field.  I also have my custom type set to 'custom field 1', which is apparently the same 'custom field' type as 'Application' which is already used:

             

            01 - event.jpg

            • 3. Re: Data enrichment clarification - where the enrichment field is added
              Scott Taschler

              - As per the 9.3.0 Product Guide p58, data acquisition takes place on the ESM, and not on the devices.  When the data acquisition takes place, are the stored events within ESM themselves enriched, and if so are all of them enriched or only events from a specific time period relevant to the acquisition, or is it the event displays that are enriched (ie the events themselves are left untouched, but when we view the events in a drill down view, the relevant custom fields are added)

               

              Only NEW events coming that come in after the enrichment is configured are enriched.  Events that are already in the ESM database are not enriched.  To clarify a bit, the process of enrichment happens in a couple of different places.  The queries to construct the enrichment lookup tables come from the ESM.  These tables are pushed to the receivers.  The actual act of doing the dynamic lookup in the table and adding fields to the events is done on the Receiver.

               

              - If I set the device to enrich to be the ePO server, will the events I am looking to enrich actually be enriched, as within the ESM events themselves, the actual device is '<EPO server> - <EPO server>_VirusScan (ePO)'.  Ie - which device should I select for this specific case? If I select the event receiver that is sending the aggregated events to ESM, will that work or not (I am assuming not).

               

              You can set the enrichment to happen at any level you prefer.  If you select a Receiver, then all events coming into that REC are enriched.  If you select "<ePO Server_VirusScan (ePO)", then only VirusScan events will be enriched.

               

              - Just for completeness, as the enrichment process is actually taking place on the ESM, I am assuming that if a specific field is missing (or may go missing) as a result of aggregation, then using it as a lookup field will not work.  This makes sense to me, but there may be things i am not taking into account - I am still fairly new to this!

               

              You are correct.

               

              Scott

              • 4. Re: Data enrichment clarification - where the enrichment field is added
                dmease729

                Cheers Scott!

                 

                Fantastic response as always - this has helped greatly!  My initial source of confusion has gone!  Now I have to figure out why my custom field isnt appearing...  I can see other posts on this which I need to read through first, but I am likely going to be following up with a new post soon!  The example that I am using is close to pointless now that I have seen that the Analyzer_DAT_Version field appears, however this is still good practise so I am going to keep at it!