8 Replies Latest reply on Apr 15, 2014 9:34 AM by kjohnson

    Max datasource rule count exceede

    oceballos

         Hi Guys

       

      I am pretty new on this SIEM, and Im dealing with one problem that I cant solve by my self.

       

      I have added a new datasource in the Local Receiver,  but it does show only one event  !Max datasource rule count exceeded", I have seen many webpages and guides but I cant find anything usefull.

       

      Does anyone have an idea on what is going on?

       

      Thanks a lot

        • 1. Re: Max datasource rule count exceede
          Scott Taschler

          It sounds like you have configured your data source with the option to "Support Generic Syslog" enabled.  This is rarely desireable, and can result in what you're seeing.  What's happening is that the Receiver is looking at the first few bytes (16 or 32...don't recall offhand) and using those as a unique identifier for a Data Source rule.  For many syslog streams we receive, the first few bytes always include a unique timestamp, so you get a new DS rule for every event, quickly filling up the DS rule table.  As new events come in, the Receiver tries to create new DS rules, but cannot, so it generates this error.

           

          You will need to disable this option on all of your data sources.  Then you will need to delete the auto-leard DS rules via the Policy Editor.  This will clear up the error condition you are currently seeing.  Then you will be able to begin troubleshooting why your events coming in from the new data source.

           

          Scott

          • 2. Re: Max datasource rule count exceede
            oceballos

            Hi Scott

             

            Thanks a lot for your reply and my apologies for my delayed reply.

             

            Context: Iḿ trying to integrate Nitro with a non out-the-box appliance, therefore I am using the Generic option in the add datasource page.

             

            I have tried to follow your instructions, but I had some questions;

             

            1)  Disable Support Generic Syslog

            -> Does it mean that I have to set the Supprt  "Do nothing"? Should I use the profiles? (in this case there is no datasource model associated and it does not allow me to create the datasource.

             

            2) Policy Editor

            I had clean the auto-learned rules.

             

            Thanks in Advance Scott

            • 3. Re: Max datasource rule count exceede
              mepplin

              As Scott mentioned, the parse as generic syslog option is good for troubleshooting, but can create an excessive amount of data source rules resulting in the policy violation you witnessed. Adding a data source in this manner will bring events into the ESM, however the rule name will be the first several bytes of the message, and the source IP is set to the IP of the data source. Other useful fields such as a source user, destination IP, etc. are not parsed out, and are therefore not searchable as fields in the ESM.

               

              The best option is to turn on the generic option for a short period of time to gather unparsed events and then create a new Advanced Syslog Parser rule or rules to properly parse the data. In the interim, simply set the option to "Log Unknown". This will group all unparseable events into a single aggregated event, making you aware that events haven't parsed, while not overloading the system with generic rules that provide little value beyond troubleshooting. Your ultimate goal should be the creation of custom ASP rules to parse the data in such a way that it is useful in your reports and searches.

               

              Mike

              • 4. Re: Max datasource rule count exceede
                oceballos

                Hi Mike, thanks for answering...

                 

                I´ve created a ASP rule for my out-of-the-box appliance syslog messages  that fits perfectly with what I see in the Data source rule when I have Support Generic Syslog activated, but when I unchek this option, there is not event at all in my dashboard, probably because it do not match correctly. any idea?

                 

                How I can say to the SIEM that an specific ASP Rules is attached to specific device? is that possible?

                 

                test3.png

                • 5. Re: Max datasource rule count exceede
                  mepplin

                  You will need to make sure the ASP rule is enabled for the policy for that device. You can find this by going to the policy editor, and to the ASP rules. Click the arrow by default policy and select your data source. Make sure the rule is enabled for that data source. It is probably disabled at that level but enabled for the default policy. You need to enable it at the data source ASP policy level and this should help. In the first image notice the highlighted rule is disabled for the DLP Policy Discover Rule. In the second image it is enabled but I am scoped to the device ASP policy. This is what you want to verify and probably enable for your parser.

                   

                  Default policy.JPG  Policy device level.JPG

                  • 6. Re: Max datasource rule count exceede
                    oceballos

                    Hi, thanks!

                    I have followed your indications, however I´m still not sure whether I´m associating the ASP rule with the device.  When I goes to the Policy editor of the device, the advanced filter puts the device id in it, so I just look for my ASP Rule called ASP, but nothing appears, just when I delete the Device type ID is when it appears...

                     

                    I have even create another rule when I had the device selected in the Policy Tree...there is another way more straigh foward to associate? it´s driving me crazy....;-)

                     

                     

                    test4.png

                    • 7. Re: Max datasource rule count exceede
                      wilson.wang

                      Yes, I met this issue too. It looks like a huge bug on Nitro..

                       

                      1) Create a new log source as a Generic Data Source

                      2) create 5 new ASPs associated with that specific data sources..

                      3) I can see events coming with a bunch of unknown events..

                      4) I drilled down few "unknown events" -> pasted the raw packets to the parser and the parse proved to be working fine with the logs.

                       

                      that means the ASP module does not parse the events properly...

                       

                      Regards

                       

                      wilson

                       

                      Message was edited by: wilson.wang on 4/14/14 10:12:08 PM CDT
                      • 8. Re: Max datasource rule count exceede
                        kjohnson

                        If you have already created custom ASP rules to parse the events, go to the rule editor and try checking the box, "Include syslog header in regular expression match". ASP attempts to match and parse the syslog header automatically. By default it is assumed that your regular expression is meant to match on the data after the header. If you check the box, then your expression will be evaluated against the entire event. This could be why your expressions match in the rule editor, but not in running policy. Also check the content strings you have defined, as the rule will not fire if the content strings don't actually appear in the log.