Any Idea please!
I'm the author of the document on how to write and troubleshoot a parser, and I plan to update this document soon to reflect changes and new features in the ESM parsing system.
One of the changes made in the ESM user interface is that data sources with no rules (what used to be referred to as just Syslog as data source vendor) is now called "Generic".
When making new parsing rules, they should be enabled and rolled out in the policy associated with your new Generic data source.
Can you please include some screenshots of what is working and not working, and I am sure we can help you.
Firstly I would like to thank you for your response
Data source config : https://www.dropbox.com/s/p2mpxi7mrirfx96/DS.PNG?dl=0
regular Expressions: Dropbox - regExp.txt
2014/09/23 23:40:23,01044,[SMTP][A7A1B41D-A283-440E-B494-C4E426ACA0D8] Forwarding mail to firstname.lastname@example.org to 220.127.116.11 at port 25
2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message from: <email@example.com>
2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message to: firstname.lastname@example.org
2014/09/23 23:40:24,03124,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Msg size=220693 bytes, processing time=78 ms, rate=2763,083 kb/s
2014/09/23 23:40:24,03580,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Forwarding mail to email@example.com to 18.104.22.168 at port 25
2014/09/23 23:57:26,03800,[SMTP] Connection from <192.168.1.5> blocked by NRS.
after the creation of custom parser, enable the rule, rolle out in the policy and upload the log file,just the first line is parsed (Forwarding mail to), the other lines are shown as unknown log file.
what can i do ?
Looking at your regex, I would suggest that when matching characters like "/", ":", "\", commas, etc, it would be best suggest to match them with the hex code for the ascii.
For example, if you're trying to match a backslash "/", you would match it with \x2f. Our PCRE can be a little temperamental when using trying to match the characters directly.
Buddy-ole-friend :-D Okay huge stretch there...... Nice document by the way, helped me a ton when I was creating some parsers.
Do you happen to have a copy of the parselog.py tool referenced on page 41 in your document?
If so, would you be willing to share that with us or point us to a website? I have googled quite a bit for that and can't seem to find it anyway on the interwebs..
Thanks a bunch,
I enclose two versions of the script which was mentioned in the document on how to write a custom parser, the original parselogs.py and a better version, improved by a customer (thank you Gene!) to use python classes and importantly, to allow more than one regular expression to match in the supplied file.
mparselog.py -r <regex_file> -l <log_file>
This script takes the following options and parameters:
-r # File containing regular expressions. One per line, no blank lines.
-l # Log File to parse.
-h # This help message.
I hope this helps!
Awesome!! This will be so useful not only in Nitroland but in general log diving. I don't see license information inside so I'll treat as such.
Thanks a bunch!
Robert (I'm guessing at your name!)
I hope it helps - do with it what you will, however it you make some improvements other may appreciate, please post it back here!
Yup... that's me