3 Replies Latest reply: Feb 11, 2013 9:40 PM by hschupp RSS

    NSM - Bulk Attack Filter Unassign?

    gabezawalski

      I've got a mess on my hands.  Two attack filters were created and added to everything instead of creating an ACL.  I want to clean it up and remove them from all the applied filters (both inbound and outbound..almost 10,000 assignments).  Is there a way to do this in bulk?  I can't imagine how much time would be wasted if there was not.

       

      Thanks.

        • 1. Re: NSM - Bulk Attack Filter Unassign?
          gabezawalski

          HA!  Figures I'd find a method as soon as I ask.

           

          1. Select all attack filters that have it applied, click bulk edit.

          2. In the new window select replace attack filters in the action drop down. 

          3. Click save.  Done. 

           

          NOTE: Do not select assignments that have additional filters to them in your bulk edit or you will lose them as well.  (Thankfully I learned this while doing the Reconnaissance and can reapply those without too much work!)

           

           

          Anyone know of another method?

           

          EDIT: even cleaner method found. on 1/10/13 10:34:12 AM CST
          • 2. Re: NSM - Bulk Attack Filter Unassign?
            hschupp

            Here is a document I wrote up for a customer who walked into a system that they had been adding filters to for years.. and now no-one knew why most of them had been created.

            This customer primarily used I-4010 or M-4050 or larger sensors which allowed them 128K to 256K of alert filters to be applied to a sensor.

             

            Chart 1: Attack Filter capabilites by sensor

            Maximum Type

            M-8000

            M-6050

            M-4050

            M-3050

            M-2950

            M-2850

            M-2750

            M-1450

            M-1250

            Customized attacks

            100,000

            100,000

            100,000

            100,000

            100,000

            100,000

            100,000

            100,000

            20,000

            Attack filters per VIDS

            128,000

            128,000

            100,000

            100,000

            100,000

            100,000

            100,000

            40,000

            20,000

            Attack filters per Sensor

            262,144

            262,144

            262,144

            262,144

            131,072

            131,072

            131,072

            65,536

            32,768

             

            But now they were deploying M-1250's out to their satellite offices.

            They found they could join the sensors to the NSM and receive the initial configuration but could not push a configuration update to the M-1250 sensors after that.

            The error told them they were exceeding the allowed attack filter count. 

             

            (joining a sensor to the NSM has fewer 'checks' for the initial configuration solely for the purpose of trying to ensure it DOES join)

             

             

             

             

             

            Anyways - here is a detailed document on manual "Filter Management" using MySQL commands.

             

            ALERT FILTER MANAGEMENT GUIDE  - USING MYSQL COMMANDS TO CLEAN UP THE NSM DATABASE

             

            On the NSM it is extremely easy to Create and Assign Alert Filters. 

            However, removing them is NOT so simple when they are applied across 100’s or 1000’s of Signatures. 

            This guide talks about how to easily craft mysql commands to remove Alert Filter Assignments manually from NSM DB using the MySQL prompt.

             

            Changes in the 7.1 Filter Management have improved the de-assignment of Alert Filters process.. but the methods below are still valid when you are trying to bulk changes.

             

            There are two parts to this document that are interwoven throughout:

             

            1. How to perform maintenance on your Attack Filters
            2. Why you should never assign an Alert Filter to ALL signatures.

             

            The instructions below speak to both issues. 

            For the “never assign a filter to all signatures” see the write-up under the “WHY YOU SHOULD NOT (EVER!) ASSIGN FILTERS TO ALL SIGNATURES’” section in the beginning of this document below.

             

            There are several levels of how to go about cleaning up the alert filters on a sensor.

             

            1. You can delete all the Filter Assignments in one fell press of the enter key
            2. You can delete all assigned filters from an Interface, or the sensor, or the Domain or a combination of them all.
            3. You can do so by deleting a particular filter from being assigned anywhere.
            4. You can delete all filters assigned to a particular attack.

             

            It all starts with ---

             

            STOP THE NSM MANAGER AND USER INTERFACE SERVICES – THEN…

             

            Log in to the mysql command prompt so you can run the queries.

            To do so open a command prompt in the <datadrive>\mysql\bin directory.

            Then type:

             

            mysql -uroot -p                                <CR>

             

            Enter your 'root' password at the prompt.

            use lf                                     <CR>

             

            Will respond with “database changed”

             

            From there you can start running queries to see how many filter assignments there are:

             

            Select count(*) from iv_alertfilterassignment;                 <CR>

             

            Will return a row count which is the total # of filter assignments.

            Do NOT let this number fool you! (see next paragraph)

                                           

            WHY YOU SHOULD NOT (EVER!) ASSIGN FILTERS TO ALL SIGNATURES:

            (especially not at the domain level!)

             

            You may get a result.. from this query above  - say it comes back with 42,645 .. and think that .. "Hey!  That is far less than the maximum!  I don’t need to clean anything out!"

            Unfortunately, that is not how it compiles when preparing the configuration to a sensor.

            The assignments are read out per Domain + per Sensor + per Interface A + per Interface B + per VID

            So, for example.. if there were 8,000 filter assignments at the domain, those would multiply as follows.. (we’ll use a I-4010 sensor that has 12 interfaces… 1A/B, 2A/B, 3A/B, 4A/B, 5A/B, 6A/B

             

            8,000   Domain

            8,000   Sensor

            96,000 Interface 8,000 x 12 interfaces

            ----------

            106,000           Alert Filter Assignments for this one sensor

             


            Now you might ask yourself how difficult is it to reach this number of filter assignments? 

            Here is an example…

             

            You decide that you want to assign a filter for your vulnerability scanner so that alerts from it are not seen in the RTTA.

            • You create a filter and assign it to all the Attacks in your policy.  The All-Inclusive with Audit Policy has approximately 4,000 signatures – in each direction (Inbound & Outbound).
            • So that is 8,000 Alert filter assignments right there.
            • You assign that to the root domain because you want it to apply to all sensors in the domain.

             

            So.. as you can see from the example above.. it only takes 1 single Alert Filter.. assigned to all signatures to suddenly have 106,000 assignments applied to a sensor.

             

            And the example there was for just an I-4010.. expand that to an M-2950 … 20 interfaces…. And you can see that this can get out of hand very quickly.

             

            The above scenario is the perfect example of why not – and, in the end, it serves to negatively impact the sensor performance having to ‘consider’ these filters for every attack generated.

             

            CAVEAT:  In the current 6.1.x.x Manager and Sensor Software code - changes were made to NOT assign filters to unused (disabled) interfaces. 

            This dramatically reduces how many filters assignments are included in a sensor configuration push.

             

             

            USE ACL’s INSTEAD

            The answer to this is to USE ACLS to do the same job. 

            Two --- yep.. Just 2!  ACLS would do the same job as assigning a filter against the entire 8000 signatures!

            You would create 1 ACL to NOT INSPECT traffic between the source and destination IPs in one direction and then 1 more for the opposite direction.

            Now.. With these two rules.. You have stopped the Vulnerability Scanner from generating the alerts in the first place.  There is no need to ‘filter’ what is never created.

            And you have saved yourself 106,000 filter assignments.

             

            Note:   When a new signature set is downloaded and applied.. the new signatures that are added are NOT going to have the filter assigned to them through any magical understanding that you had the filter assigned to all the signatures previously in the database. 

             

            So, if for example, 22 new signatures were added in the new sigset.. suddenly your vulnerability scanner would be seen generating alerts for those new signatures because the filter assignments did not include them.   You would have to constantly keep updating the filter assignments with every sigset release.

             

            The ACL will not be affected by this.. so it is a Set and Forget option that performs the same function for far less work and no follow-up maintenance.

             

            NOTE1: BACK UP YOUR DATABASE BEFORE PERFORMING ANY OF THE FOLLOWING DELETION SCRIPTS!

             

            THE “BARE KNUCKLE – IT’S A KO!!!” SCRIPT

             

            To simply delete all alert filter assignments you can do the following command:

             

            Delete from iv_alertfilterassignment;         <CR>

            Will delete all filter assignments from the database

            There is no recovery.. it’s down for the count.

            (untruth – a restore from a config-tables or all-tables backup will bring them back)

             

            NOTE2: BACK UP YOUR DATABASE BEFORE PERFORMING THIS OR THE FOLLOWING SCRIPTS!

             

             

            THE “KID GLOVES - GENTLER HAND…” SCRIPTS

             

            However, if you want to delete only those alerts assigned to a particular sensor you can first run the following queries to give you the variables for creating some very specific queries depending on your needs.

            Run each of the following commands and copy the results to notepad.

             

            GET ALL SENSOR NAMES AND IDS:

            Select sensor_id, name from iv_sensor;      <CR>

             

            Will display all the sensor names and their associated “ID”

            Copy this out to notepad or a spreadsheet

             

            GET ALL SENSOR INTERFACE NAMES AND IDS:

            Select sensor_id, vids_id, name from iv_vids;        <CR>

             

            Will display the sensor_ID (refer to query above) and the Interface/VIDS IDs

            Useful for queries to delete alert filters at a particular sensor interface.

             

            GET ALL FILTER NAMES AND IDS:

            Select group_id, group_name from iv_alertfiltergroup;   <CR>

             

            Will display all the attack filter names and the associated group ID for that filter

             

            GET ALL DOMAIN NAMES AND IDS:

            Select subscriber_id, name from iv_subscriber;     <CR>

             

            Will display domain and child domain Ids that you have created

            Useful for queries to delete alert filters at a particular domain level only.

             

            We will use the variables from these queries to craft new deletion queries that only delete specific sets of filter assignments:

             

             

            Now we’ll look at how to use these commands to remove the Alert Filter Assignments. 

             

             

             

             

            Continued in Next Reply

             

            Message was edited by: hschupp on 2/11/13 10:37:26 PM EST
            • 3. Re: NSM - Bulk Attack Filter Unassign?
              hschupp

              Now that you have collected all that information -- here are the scripts to use that data to start removing the excess filters from the database.

               

               

              For example, to delete all filters assigned to a particular sensor you would find the sensor name in the output of the first of the queries above and then type the command:

               

              DELETE ALL FILTERS ASSIGNED TO A SINGLE SENSOR:

               

                              Delete from iv_alertfilterassignment where sensor_id=1002;  <CR>

               

              Will delete all filter assignments from the database for just that one sensor.

               

              Or, how about deleting all assignments for a particular filter from the Database?

              Is great for when you want to delete a filter from the GUI but can’t find all the places it is assigned to:

               

              DELETE AN INDIVIDUAL FILTER FROM BEING ASSIGNED TO ANYTHING:

               

              Delete from iv_alertfilterassignment where group_id=301;         <CR>

               

              Will delete all filter assignments from the database for just that one filtername.

              The group_id is found in the second of the two Select queries from above.

               

              Maybe you just want to delete the queries for a child domain in order to clean up filters (or perhaps you want to delete the child domain but can’t because there are filters assigned to it).

               

              DELETE ALL FILTERS ASSIGNED TO THE ROOT DOMAIN:

               

              Delete from iv_alertfilterassignment where subscriber_id=0;      <CR>

               

              Will delete all filter assignments from the database for the root domain.

               

              DELETE ALL FILTERS ASSIGNED TO A CHILD DOMAIN:                      

               

              Delete from iv_alertfilterassignment where subscriber_id=104;  <CR>

               

              Will delete all filter assignments from the database for that child domain.

               

              You can combine these to get even more refined:

              DELETE ALL FILTERS ASSIGNED TO A SINGLE SENSOR INTERFACE IN A CHILD DOMAIN:

               

              Delete from iv_alertfilterassignment where subscriber_id=104 and sensor_id=1003 and iv_vids=213; <CR>

               

              Will delete all filter assignments from the database for that child domain for only that sensor and only on that interface.

               

              And… as mentioned in the very beginning you can delete all filter assignments to a specific signature.

              This requires looking up the Attack ID for the signature first and then plugging that into the following query:

               

              Delete from iv_alertfilterassignment where attack_id=<attack_id>

               

              For example the attack_id for the signature “HTTP: HTTP HTML PAGE NOT FOUND” attack is 0x40280200.  The command to delete any filter assignments on that attack would be:

               

              DELETE ALL FILTERS ASSIGNED TO A SINGLE ATTACK ID:

               

              Delete from iv_alertfilterassignment where attack_id=’0x40280200’;                    <CR>

               

              Will delete all filter assignments from the database for that child domain for only that sensor and only on that interface.

               

              Note the use of single quotes around the attack_id value. This had not been necessary in the previous queries where the value was just a decimal value.

               

               

              THE “REFEREE’S DECISION” – SUMMARY

               

              These should give you a good handle how to craft deletion queries on the iv_alertfilterassignment table to help get your filters under control.

               

              In the basest of terms – if you do not want to detect alerts between a particular source and destination

              (which is really all a filter defines) then use an ACL to tell the sensor not to inspect that traffic. 

              That just takes TWO ACL rules instead of 8000 filter assignments.  In addition, the ACL is FAR less impactful on sensor performance

               

              Always yours in support,

               

              Henry “Hank” Schupp

               

              Did I mention?:   BACK UP YOUR DATABASE BEFORE PERFORMING ANY OF THE PREVIOUS DELETION SCRIPTS!

              <grin>