We've got an in-house application that i've written quite a regex for and it's all worked wonderfully until yesterday. The application has been inserting errors in to the syslog stream when it has issues connecting to an external provider source. Originally, I was told that we didn't need to track those errors so I didn't need to write any regex for it and we just dropped those events as unknown. Yesterday, the applications team decided it would be good to start tracking these events so I created a very simple set of regex for it. I loaded it in to the SIEM, rolled out my policy editor and went home for the day a few hours later and not thinking about it until this morning.
I checked the events and they all stopped right when I loaded the new rule. Did my basic troubleshooting of NitroStop/Start --nod and still nothing. I also noticed at this time that my WMI parsing was at a grinding halt. So I correlated the times that I rolled out the new rule with the last time I got the events. They all matched, so I disabled the rule and everything starts working again. So i'm a bit confused why a simple rule that perfectly matches the event would kill everything so spectacularly.
Here is a sample event and the regex that I used. Any ideas?
<163>1 2017-07-10T12:16:28.266943-05:00 APPLICATIONSERVER-01 APP1 6168 31 - Error connecting to ServiceProvider
I've obviously had to change some information in the event to keep our data private, but it's all the same format as the actual event and still parses as far as I can see. The only thing I can think of is that it's too simple? Maybe I should be utilizing context strings in the regex to be much more specific and there by reducing any requirements for verbosity that the parser might choke on?
Any help would be greatly appreciated! Thanks in advance!
No unknown events or anything, just a complete stop on all ASP parsing and a good portion of WMI. The moment I disabled that rule and rolled out, it all started working again.