It sounds like you have configured your data source with the option to "Support Generic Syslog" enabled. This is rarely desireable, and can result in what you're seeing. What's happening is that the Receiver is looking at the first few bytes (16 or 32...don't recall offhand) and using those as a unique identifier for a Data Source rule. For many syslog streams we receive, the first few bytes always include a unique timestamp, so you get a new DS rule for every event, quickly filling up the DS rule table. As new events come in, the Receiver tries to create new DS rules, but cannot, so it generates this error.
You will need to disable this option on all of your data sources. Then you will need to delete the auto-leard DS rules via the Policy Editor. This will clear up the error condition you are currently seeing. Then you will be able to begin troubleshooting why your events coming in from the new data source.
Thanks a lot for your reply and my apologies for my delayed reply.
Context: Iḿ trying to integrate Nitro with a non out-the-box appliance, therefore I am using the Generic option in the add datasource page.
I have tried to follow your instructions, but I had some questions;
1) Disable Support Generic Syslog
-> Does it mean that I have to set the Supprt "Do nothing"? Should I use the profiles? (in this case there is no datasource model associated and it does not allow me to create the datasource.
2) Policy Editor
I had clean the auto-learned rules.
Thanks in Advance Scott
As Scott mentioned, the parse as generic syslog option is good for troubleshooting, but can create an excessive amount of data source rules resulting in the policy violation you witnessed. Adding a data source in this manner will bring events into the ESM, however the rule name will be the first several bytes of the message, and the source IP is set to the IP of the data source. Other useful fields such as a source user, destination IP, etc. are not parsed out, and are therefore not searchable as fields in the ESM.
The best option is to turn on the generic option for a short period of time to gather unparsed events and then create a new Advanced Syslog Parser rule or rules to properly parse the data. In the interim, simply set the option to "Log Unknown". This will group all unparseable events into a single aggregated event, making you aware that events haven't parsed, while not overloading the system with generic rules that provide little value beyond troubleshooting. Your ultimate goal should be the creation of custom ASP rules to parse the data in such a way that it is useful in your reports and searches.
Hi Mike, thanks for answering...
I´ve created a ASP rule for my out-of-the-box appliance syslog messages that fits perfectly with what I see in the Data source rule when I have Support Generic Syslog activated, but when I unchek this option, there is not event at all in my dashboard, probably because it do not match correctly. any idea?
How I can say to the SIEM that an specific ASP Rules is attached to specific device? is that possible?
You will need to make sure the ASP rule is enabled for the policy for that device. You can find this by going to the policy editor, and to the ASP rules. Click the arrow by default policy and select your data source. Make sure the rule is enabled for that data source. It is probably disabled at that level but enabled for the default policy. You need to enable it at the data source ASP policy level and this should help. In the first image notice the highlighted rule is disabled for the DLP Policy Discover Rule. In the second image it is enabled but I am scoped to the device ASP policy. This is what you want to verify and probably enable for your parser.
I have followed your indications, however I´m still not sure whether I´m associating the ASP rule with the device. When I goes to the Policy editor of the device, the advanced filter puts the device id in it, so I just look for my ASP Rule called ASP, but nothing appears, just when I delete the Device type ID is when it appears...
I have even create another rule when I had the device selected in the Policy Tree...there is another way more straigh foward to associate? it´s driving me crazy....;-)
Yes, I met this issue too. It looks like a huge bug on Nitro..
1) Create a new log source as a Generic Data Source
2) create 5 new ASPs associated with that specific data sources..
3) I can see events coming with a bunch of unknown events..
4) I drilled down few "unknown events" -> pasted the raw packets to the parser and the parse proved to be working fine with the logs.
that means the ASP module does not parse the events properly...
If you have already created custom ASP rules to parse the events, go to the rule editor and try checking the box, "Include syslog header in regular expression match". ASP attempts to match and parse the syslog header automatically. By default it is assumed that your regular expression is meant to match on the data after the header. If you check the box, then your expression will be evaluated against the entire event. This could be why your expressions match in the rule editor, but not in running policy. Also check the content strings you have defined, as the rule will not fire if the content strings don't actually appear in the log.