cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Highlighted

Malfunction of Syslog custom parser

Jump to solution

Hi,

I have followed the following procedure to configure a custom parser :

Dropbox - How to write a McAfee ESM Custom Parser and troubleshoot a data source.pdf


I have configured a custom Parser for the InterScan Data source, i did not find Syslog as Datasource vendor  as mentioned in the procedure above, so i have used these parameters :

--> Data source vendor : Trend Micro

--> Data source model: Interscan web security suite (ASP)


Data source config : Dropbox - DS.PNG


I relied on a sample log file and I have configured  5 regular Expressions:


2014/09/23 23:40:23,01044,[SMTP][A7A1B41D-A283-440E-B494-C4E426ACA0D8] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message from: <exemple@domaine.com>

2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message to: exemple@domaine.com

2014/09/23 23:40:24,03124,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Msg size=220693 bytes, processing time=78 ms, rate=2763,083 kb/s

2014/09/23 23:40:24,03580,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

2014/09/23 23:57:26,03800,[SMTP][] Connection from <192.168.1.5> blocked by NRS.

..

..

..

If we test the parser locally (sample log data zone), it can parse and extract useful information from the log files :

Dropbox - parsing.PNG


But, after upload the sample log file, just the first line is parsed (Forwarding mail to), the other lines are shown as unknown log file.


And today i have see this error :  Dropbox - parser_con.PNG


Is it mandatory to use Syslog as Data source Vendor ?


I count on your reactivity, I am currently blocked!


Best Regards

Hi,
We have configured a custom Parser for the InterScan Data source, we did not find Syslog as Datasource vendor (Enclosed a screenshot : 1).
If we test the parser locally, it can parse and extract useful information from the log files (screenshot 2), by against it uses only the first regular expression to handle log files received from the server.
Is it mandatory to use Syslog as Data source Vendor ?
Enclosed a sample log data (InterscanLog).

We count on your reactivity
Best Regards
1 Solution

Accepted Solutions
Highlighted

Re: Malfunction of Syslog custom parser

Jump to solution


Hi,

Firstly I would like to thank you for your response

Data source config : https://www.dropbox.com/s/p2mpxi7mrirfx96/DS.PNG?dl=0

regular Expressions: Dropbox - regExp.txt


Log file:



2014/09/23 23:40:23,01044,[SMTP][A7A1B41D-A283-440E-B494-C4E426ACA0D8] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message from: <exemple@domaine.com>

2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message to: exemple@domaine.com

2014/09/23 23:40:24,03124,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Msg size=220693 bytes, processing time=78 ms, rate=2763,083 kb/s

2014/09/23 23:40:24,03580,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

2014/09/23 23:57:26,03800,[SMTP][] Connection from <192.168.1.5> blocked by NRS.

after the creation of custom parser, enable the rule, rolle out in the policy and upload the log file,just the first line is parsed (Forwarding mail to), the other lines are shown as unknown log file.


what can i do ?


Best regards,


View solution in original post

10 Replies
Highlighted

Re: Malfunction of Syslog custom parser

Jump to solution

Hi,

Any Idea please!

BR

Highlighted
Level 7
Report Inappropriate Content
Message 3 of 11

Re: Malfunction of Syslog custom parser

Jump to solution

Hello,

I'm the author of the document on how to write and troubleshoot a parser, and I plan to update this document soon to reflect changes and new features in the ESM parsing system.

One of the changes made in the ESM user interface is that data sources with no rules (what used to be referred to as just Syslog as data source vendor) is now called "Generic".

When making new parsing rules, they should be enabled and rolled out in the policy associated with your new Generic data source.

Can you please include some screenshots of what is working and not working, and I am sure we can help you.

Best regards

Ian

Highlighted

Re: Malfunction of Syslog custom parser

Jump to solution


Hi,

Firstly I would like to thank you for your response

Data source config : https://www.dropbox.com/s/p2mpxi7mrirfx96/DS.PNG?dl=0

regular Expressions: Dropbox - regExp.txt


Log file:



2014/09/23 23:40:23,01044,[SMTP][A7A1B41D-A283-440E-B494-C4E426ACA0D8] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message from: <exemple@domaine.com>

2014/09/23 23:40:24,03344,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Message to: exemple@domaine.com

2014/09/23 23:40:24,03124,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Msg size=220693 bytes, processing time=78 ms, rate=2763,083 kb/s

2014/09/23 23:40:24,03580,[SMTP][C13827D2-EFB9-46CB-9556-28EBD3DDCD67] Forwarding mail to exemple@domaine.com to 200.200.200.23 at port 25

2014/09/23 23:57:26,03800,[SMTP][] Connection from <192.168.1.5> blocked by NRS.

after the creation of custom parser, enable the rule, rolle out in the policy and upload the log file,just the first line is parsed (Forwarding mail to), the other lines are shown as unknown log file.


what can i do ?


Best regards,


View solution in original post

Highlighted
McAfee Employee
McAfee Employee
Report Inappropriate Content
Message 5 of 11

Re: Malfunction of Syslog custom parser

Jump to solution

Looking at your regex, I would suggest that when matching characters like "/", ":", "\", commas, etc, it would be best suggest to match them with the hex code for the ascii.

For example, if you're trying to match a backslash "/", you would match it with \x2f. Our PCRE can be a little temperamental when using trying to match the characters directly.

Highlighted
Level 7
Report Inappropriate Content
Message 6 of 11

Re: Malfunction of Syslog custom parser

Jump to solution

it seems not working for me. can you have a look my regex and log?

log: <14>Jul 11 14:48:53 hostname 1,2018/07/11 14:48:53,013201001919,SYSTEM,url-filtering,0,2018/07/11 14:48:53,,upgrade-url-database-success,,0,0,general,informational,"PAN-DB was upgraded to version 20180711.20225.",6469838288294402439,0x0,0,0,0,0,,hostname

regex: (\d+)\>(\w+\s+\d+\s+\d+\x3a\d+\x3a\d+)\s+(\w+)\s+\d+\x2c(\d+\x2f\d+\x2f\d+\s+\d+\x3a\d+\x3a\d+)\x2c(\d+)\x2c(\w+)\x2c(\w.+)\x2c\d+\x2c(\d+\x2f\d+\x2f\d+\s+\d+\x3a\d+\x3a\d+)\x2c\x2c(\w.+)\x2c\x2c+\d\x2c\d\x2c(\w+)\x2c(\w+)\x2c\x22(\w.+)\x22\x2c(\d+)\x2c\d+\x\d+\x2c\d+\x2c\d+\x2c\d+\x2c\d+\x2c\x2c(\w+)

Highlighted
Level 9
Report Inappropriate Content
Message 7 of 11

Re: Malfunction of Syslog custom parser

Jump to solution

Ian,

Buddy-ole-friend 😄    Okay huge stretch there......  Nice document by the way, helped me a ton when I was creating some parsers.

Do you happen to have a copy of the parselog.py tool referenced on page 41 in your document?

If so, would you be willing to share that with us or point us to a website?  I have googled quite a bit for that and can't seem to find it anyway on the interwebs..

Thanks a bunch,

  -B

Highlighted
Level 7
Report Inappropriate Content
Message 8 of 11

Re: Re: Malfunction of Syslog custom parser

Jump to solution

Hi there,

I enclose two versions of the script which was mentioned in the document on how to write a custom parser, the original parselogs.py and a better version, improved by a customer (thank you Gene!) to use python classes and importantly, to allow more than one regular expression to match in the supplied file.

Original:

parselog.jpg

Improved:

mparselog.py -r <regex_file> -l <log_file>

This script takes the following options and parameters:

mparselog.py

        -r            # File containing regular expressions. One per line, no blank lines.

        -l            # Log File to parse.

        -h            # This help message.

I hope this helps!

Regards

Ian

Highlighted
Level 9
Report Inappropriate Content
Message 9 of 11

Re: Malfunction of Syslog custom parser

Jump to solution

Ian,

  Awesome!!  This will be so useful not only in Nitroland but in general log diving.   I don't see license information inside so I'll treat as such.

Thanks a bunch!

-B

Highlighted
Level 7
Report Inappropriate Content
Message 10 of 11

Re: Malfunction of Syslog custom parser

Jump to solution

Robert (I'm guessing at your name!)

I hope it helps - do with it what you will, however it you make some improvements other may appreciate, please post it back here!

thanks

Ian

You Deserve an Award
Don't forget, when your helpful posts earn a kudos or get accepted as a solution you can unlock perks and badges. Those aren't the only badges, either. How many can you collect? Click here to learn more.

Community Help Hub

    New to the forums or need help finding your way around the forums? There's a whole hub of community resources to help you.

  • Find Forum FAQs
  • Learn How to Earn Badges
  • Ask for Help
Go to Community Help

Join the Community

    Thousands of customers use the McAfee Community for peer-to-peer and expert product support. Enjoy these benefits with a free membership:

  • Get helpful solutions from McAfee experts.
  • Stay connected to product conversations that matter to you.
  • Participate in product groups led by McAfee employees.
Join the Community
Join the Community