9 Replies Latest reply on Nov 14, 2014 11:49 AM by rcavey

    Malfunction of Syslog custom parser

    abdessamad

      Hi,

      I have followed the following procedure to configure a custom parser :

      Dropbox - How to write a McAfee ESM Custom Parser and troubleshoot a data source.pdf


      I have configured a custom Parser for the InterScan Data source, i did not find Syslog as Datasource vendor  as mentioned in the procedure above, so i have used these parameters :

      --> Data source vendor : Trend Micro

      --> Data source model: Interscan web security suite (ASP)


      Data source config : Dropbox - DS.PNG


      I relied on a sample log file and I have configured  5 regular Expressions:


      2014/09/23 23:40:23,01044,[SMTP][A7A1B41D-A283-440E-B494-C4E426ACA0D8] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

      2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message from: <exemple@domaine.com>

      2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message to: exemple@domaine.com

      2014/09/23 23:40:24,03124,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Msg size=220693 bytes, processing time=78 ms, rate=2763,083 kb/s

      2014/09/23 23:40:24,03580,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

      2014/09/23 23:57:26,03800,[SMTP][] Connection from <192.168.1.5> blocked by NRS.

      ..

      ..

      ..

       

      If we test the parser locally (sample log data zone), it can parse and extract useful information from the log files :

      Dropbox - parsing.PNG


      But, after upload the sample log file, just the first line is parsed (Forwarding mail to), the other lines are shown as unknown log file.


      And today i have see this error :  Dropbox - parser_con.PNG


      Is it mandatory to use Syslog as Data source Vendor ?


      I count on your reactivity, I am currently blocked!


      Best Regards

        • 1. Re: Malfunction of Syslog custom parser
          abdessamad

          Hi,

          Any Idea please!

          BR

          • 2. Re: Malfunction of Syslog custom parser

            Hello,

             

            I'm the author of the document on how to write and troubleshoot a parser, and I plan to update this document soon to reflect changes and new features in the ESM parsing system.

             

            One of the changes made in the ESM user interface is that data sources with no rules (what used to be referred to as just Syslog as data source vendor) is now called "Generic".

             

            When making new parsing rules, they should be enabled and rolled out in the policy associated with your new Generic data source.

             

            Can you please include some screenshots of what is working and not working, and I am sure we can help you.

             

            Best regards

             

            Ian

            • 3. Re: Malfunction of Syslog custom parser
              abdessamad


              Hi,

              Firstly I would like to thank you for your response

              Data source config : https://www.dropbox.com/s/p2mpxi7mrirfx96/DS.PNG?dl=0

              regular Expressions: Dropbox - regExp.txt


              Log file:



              2014/09/23 23:40:23,01044,[SMTP][A7A1B41D-A283-440E-B494-C4E426ACA0D8] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

              2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message from: <exemple@domaine.com>

              2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message to: exemple@domaine.com

              2014/09/23 23:40:24,03124,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Msg size=220693 bytes, processing time=78 ms, rate=2763,083 kb/s

              2014/09/23 23:40:24,03580,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

              2014/09/23 23:57:26,03800,[SMTP][] Connection from <192.168.1.5> blocked by NRS.

               

              after the creation of custom parser, enable the rule, rolle out in the policy and upload the log file,just the first line is parsed (Forwarding mail to), the other lines are shown as unknown log file.


              what can i do ?


              Best regards,


               

              • 4. Re: Malfunction of Syslog custom parser
                Richard Hart

                Looking at your regex, I would suggest that when matching characters like "/", ":", "\", commas, etc, it would be best suggest to match them with the hex code for the ascii.

                 

                For example, if you're trying to match a backslash "/", you would match it with \x2f. Our PCRE can be a little temperamental when using trying to match the characters directly.

                • 5. Re: Malfunction of Syslog custom parser
                  rcavey

                  Ian,

                   

                  Buddy-ole-friend :-D    Okay huge stretch there......  Nice document by the way, helped me a ton when I was creating some parsers.

                   

                  Do you happen to have a copy of the parselog.py tool referenced on page 41 in your document?

                   

                  If so, would you be willing to share that with us or point us to a website?  I have googled quite a bit for that and can't seem to find it anyway on the interwebs..

                   

                  Thanks a bunch,

                    -B

                  • 6. Re: Re: Malfunction of Syslog custom parser

                    Hi there,

                     

                    I enclose two versions of the script which was mentioned in the document on how to write a custom parser, the original parselogs.py and a better version, improved by a customer (thank you Gene!) to use python classes and importantly, to allow more than one regular expression to match in the supplied file.

                     

                    Original:

                    parselog.jpg

                    Improved:

                     

                    mparselog.py -r <regex_file> -l <log_file>

                     

                    This script takes the following options and parameters:

                     

                    mparselog.py

                            -r            # File containing regular expressions. One per line, no blank lines.

                            -l            # Log File to parse.

                            -h            # This help message.

                     

                    I hope this helps!

                     

                    Regards

                     

                    Ian

                    • 7. Re: Malfunction of Syslog custom parser
                      rcavey

                      Ian,

                       

                        Awesome!!  This will be so useful not only in Nitroland but in general log diving.   I don't see license information inside so I'll treat as such.

                       

                      Thanks a bunch!

                      -B

                      • 8. Re: Malfunction of Syslog custom parser

                        Robert (I'm guessing at your name!)

                         

                        I hope it helps - do with it what you will, however it you make some improvements other may appreciate, please post it back here!

                         

                        thanks

                         

                        Ian

                        • 9. Re: Malfunction of Syslog custom parser
                          rcavey

                          Yup... that's me

                           

                          Will do!!

                           

                          Cheers,

                            -B