Weird syslog parsing issue causing gibberish events to show up

[anhp]

Hello,

I was trying to write a custom parser for one of our applications (ManageEngine Password Manager Pro) to collect syslog and once I imported a test file, the event does show up correctly, along with like another 10 gibberish events. The regex expression I'm using for my parser is as below:

(.*?):(\d*.\d*.\d*.\d) ([a-zA-Z\_]*) \d*\/\d*\/\d* \d*:\d*:\d* ([a-zA-Z]*) (.*) ([a-zA-Z0-9-]*):(.*):(.*)

My sample syslog:

admin:127.0.0.1 Account_Added 2009/12/23 11:39:00 Success pmp_test windows-server1:account1:Testing

Using this regex, I was able to capture everything, except the date and time which I don't need. When I finished rolling out the parser to the data source and imported the test log file, I got the events, but also received weird gibberish like below:

Does anybody know why these are being captured?

Thanks,

Anh Pham

[itgfcsys]

AP

First, I've not yet tried a parser, but I do have a dosey coming up.

I might check where your parser is configured and the data sources it is applied to. Also you might take a look at the generic syslog setting in the Data source config.

Rick

[andy777 · Accepted solution]

Without looking too closely at your regex I'll say that's a lot of .*'s and ask you to try this:

(\w+)\x3a((?:\d{1,3}\x2e){3}\d{1,3})\s([^\s]+)\s(\d+\x2f\d+\x2f\d+\s\d+\x3a\d+\x3a\d+)\s(\w+)\s([^x3a]+)\x3a(\w+)\x3a(\w+)

Also make sure 'Log Unknown Syslog Event' is selected under the data source and not 'Parse as generic syslog', which should basically never be enabled, ever.

[anhp]

Thanks guys for responding!

: Having "Parse as generic syslog" on was indeed my problem. I changed it to "Log Unknown Syslog Event" and the gibberish is gone. Now that we're talking about this though, when should I have "Parse as generic syslog" enabled? I'm asking because I was working with a consultant partner with Intel and he helped me develop this parser and basically said the opposite, that I should have "Parse as generic syslog" on and should never have "Log Unknown Syslog Event" on, and when he saw the gibberish, he couldn't explain why.... so much for McAfee partner (Allen Corp)...

Also, thanks for the regex clean-up!

[andy777]

Parse generic syslog existed long before Log Unknown and really probably should have been removed when Log Unknown was added. You will not have a reason to enable Parse as generic, and now that you've had it enabled, you'll want to go into your Policy Manager and delete the autolearned rules (Data Source then under Edit on the menu).

Use Log Unknown when working on parsers. Another tip would be to have the logs collected on a syslog server until your parser is done so you can test it with a larger sample.

[itgfcsys]

Andy

Thank you for the very concise and helpful response. I've copied it off to a notepad for future reference on an upcoming parser project, Physical security badge system logs.

Rick

[anhp]

Rick,

I'm actually about to start on a parser for a physical security badge system as well, what product are you using for your badging solution?

Thanks,

AP

[itgfcsys]

AP

Lenel onguard, which one are you starting?

Rick

[anhp]

Rick,

Oh, that looks way better than ours for sure, ours is called DSX Access System (DSX Access Systems, Inc.). Just thought there was a slight chance of us wring the parser together haha..

Thanks,

AP

[itgfcsys]

AP

That would have been nice, Good luck.

Rick