8 Replies Latest reply on Sep 22, 2015 5:55 AM by itzamlan

    Is only one regex need to be deployed for parsing all kinds of logs?

    itzamlan

      I am very new to the appliance. And have not tried deploying any regex for parsing the log data. My question is, I have four different databases : Netezza, postgres, SQLServer and Oracle. And all these four dbs have different log formats. So is it like, I have to write a single regex which could work for all these different kinds of logs. Or can I put mulitple regexes based on the format of the logs?

      I attended a demo session for the same, and the presenter showed me only one console available for putting the regex. So is it the only one console available for all the regexes? Then how do I split the regexes? What would be delimiter in this case?

      For Oracle, and SQLServer I have heard like there is out of box support available. So How do I use it?

      Once again.., since I am very new to this appliance, need to understand the basics of it. In the mean time I have gone through several docs, but I was not able to catch the no. of regexes that needs to be deployed for parsing a variety of logs.

       

      Thanks in advance.

        • 1. Re: Is only one regex need to be deployed for parsing all kinds of logs?
          rhinomike

          itzamlan

           

          No. You don't need a single regex. In fact most supported devices will have more than 1 regex, far more...  help

           

          This document may help

           

          Good luck

          • 2. Re: Is only one regex need to be deployed for parsing all kinds of logs?
            itzamlan

            rhinomike

            It happened to be like, I have gone through the doc mentioned by you.

            But, here my question is what delimiter do we use to differentiate between two different regexes? Let it for a db we receive more than one format in the logs, so we put multiple regexes based on our need. So here do we put all the regexes in the same console? And if so, how do we differentiate?

            • 3. Re: Is only one regex need to be deployed for parsing all kinds of logs?
              rhinomike

              itzamlan

               

              It is up to you... let say you have 2 log line like:

               

              May 11 10:40:48 scrooge oracle-health.bin[26783]: [ID 702911 user.error] m:SY-mon-full-500 c:H : partition health measures for /var did not suffice - still using 96% of partition space

              May 11 10:40:49 scrooge pgsql[26783]: [ID 702911 user.error] partition health measures for /var did not suffice - still using 96% of partition space

               

              You could create a single rule with a regex like:

               

              (?<db_type>oracle-health.bin|pgsql)\x5b\d+\x5d\x3a\s+\x5bID\s\d+\s([a-z\.]+)\x5d (?:\sm\x3a(?<message_id>SY-mon-full-500)|).*\spartition\shealth\smeasures\sfor\s (?<partition_name>\S+)\sdid\snot\ssuffice\s-\sstill\susing\s(?<usedspace>\d+)\x2 5\sof\spartition\sspace

               

              Where I use the pipe character "|" to create conditionals to differentiate between the two type of logs: (In PCRE the pipe works like sort of like a CASE statement)


              (?<message_id>SY-mon-full-500)|)

               

              Or you could for example create two rules:

               

              One with:

               

              pgsql\x5b\d+\x5d\x3a\s+\x5bID\s\d+\s([a-z\.]+)\x5d\spartition\shealth\smeasures\ sfor\s(?<partition_name>\S+)\sdid\snot\ssuffice\s-\sstill\susing\s(?<usedspace>\ d+)\x25\sof\spartition\sspace

               

              and another one with:

               

              oracle-health.bin\x5b\d+\x5d\x3a\s+\x5bID\s\d+\s([a-z\.]+)\x5d\sm\x3a(?<message_ id>SY-mon-full-500).*\spartition\shealth\smeasures\sfor\s(?<partition_name>\S+)\ sdid\snot\ssuffice\s-\sstill\susing\s(?<usedspace>\d+)\x25\sof\spartition\sspace

               

              Had it been me, I would probably create two different rules, one for each technology provider.

               

              The main reason for that is that as you correctly noticed, the SIEM comes with a number of different rules out of the box. These are group by product/vendor IDs that help you selecting a system to match SQL Server or any other system supported out of the box.

              • 4. Re: Is only one regex need to be deployed for parsing all kinds of logs?
                jrybicki

                itzamlan,

                 

                     The ESM supports Oracle and SQLServer without the need for custom parsers. That is to say that you can create a data source on your receiver, provide it login credentials and information needed to login to the dB, and it will be able to pull some events from the database.  The ESM by default will read the audit tables in this situation (you have to have dB auditing turned on on the dB for this to work) and you will see events such as user logins to your dB and some of the other Audit Events. The information you will get from these types of pulls is useful, but don't expect so see everything that goes on.  You wont see commands that were run on the dB, SQL injection attacks etc.  If you wish to get that type of granularity take a look at the DEM (DSM) for the SIEM or look at a Database Access Management product. These products can get you more information about whats happening on your dBs. 

                 

                     In regards to the custom parser for the unsupported dB type; the first hurdle is getting the information into the ESM.  You can use a generic log source type, but you will have to find a way of writing what you need to a file that can be ingested by the ESM (via, scp, ftp, etc.)  Once you have that problem solved, you can move on to creating a custom parser.  If the format is consistent for all your events, then you should be able to get away with creating one regex expression to parse all your logs.  However, if it changes, you will need to build multiple expressions to fit the various log formats.  The custom parser is fairly intuitive, and you can paste a sample log in the editor to ensure that it grabs the intended fields. Once you have the regex built, assign the output to the fields and you should be good to go. 

                 

                I hope this helps.  I can elaborate if you need further information. 

                • 5. Re: Is only one regex need to be deployed for parsing all kinds of logs?
                  itzamlan

                  jrybicki

                  Thanks for sharing the information. But you have mentioned it like the receiver will be able to pull some of the events from the database. In my project I need to track the occurrences of all grant events on four different DBs. So by default there is out of box support for all grant events for SQLServer and Oracle right?.. I just need to create the data source, provide the credentials and turn on the auditing. That's it right?

                  • 7. Re: Is only one regex need to be deployed for parsing all kinds of logs?
                    jrybicki

                    itzamlan,

                     

                         I my experience (and I am not a DBA nor have even been one), so long as what you are looking for exists or can exist on the Audit Table, you should be able to pull in that information.  If that cannot exist in the Audit Table, then you would either have to exists as syslog (and Oracle Syslog is pretty terse), or you would have to find some way of importing it into the ESM and parsing it through a custom parser.    I imagine that this type of information is in the Audit Table, but I dot know for sure.  If you are really concerned about writes, privileges, etc,  you might want to look at something like McAfee DAM (McAfee Data Center Security Suite for Databases | Intel Security Products) This will get you all sorts if good information and is easily integrated into the ESM.