3 Replies Latest reply on Mar 24, 2008 5:54 PM by cessnai

    Rogue System start/stop events overloading system

      I'm currently investigating a problem in a 3.6.1 environment (latest server, latest agent, mix of 8.0 and 8.5 installations with anti-spyware module).

      We're experiencing problems where the Events directory on the EPO suddenly fills up with PKG files, it can be around 1000 to 2000 files in queue. EPO logs indicates that the files are taken care of, but it takes a lot of time.

      The Rogue System Sensor has been identified as the problem source. For some reason, on some clients it suddenly starts generating a MASSIVE amount of events of type 12000 and 12002 (ie Sensor Started and Sensor Stopped), at the rate of one per second or even more. Those events ends up in the local agent event directory, and at the next ASCI the agent starts sending them to the EPO server, 100 at a time. This in turn overloads the EPO server.

      So far we've identified six sensors (out of roughly 250) that behaves this way, but we can't figure out why. Sensor logs says nothing useful, neither do event logs or any other logs. Restarting the sensor server seems to "fix" the problem.


      Any clues? Turning of those events through the eventfltr.ini file will probably fix the problem on the EPO level, but not locally. Sensor version is the latest.
        • 1. RE: Rogue System start/stop events overloading system
          tonyb99
          what is your sensor config set to?
          • 2. RE: Rogue System start/stop events overloading system
            Policy-General is default, in Binding all checkboxes are ticked and two (different) networks are specified in the edit boxes.

            RSD config is pretty much default, sensor timeout is 90 minutes, max primaries per subnet is 2 and max active time is 12 hours. No grace period.

            Oh, and the clients are standard XP machines with 512 or 1 GB memory.

            This is what ends up in the local sensor log:

            03-19-08 07:49:00, [1924] INFO RSSENSOR <> - Virtual sensor 1924 initialized at network address: xxx.yyy.zzz.0 on interface 'Intel(R) PRO/100 VE Network Connection (Microsoft's Packet Scheduler)

            03-19-08 07:49:00, [1924] INFO RSSENSOR.ServerCom <> - Queueing host detection message for later transmission, due to sensor throttle

            ...and then it just repeats those two lines, about 7-8 times per second.


            I ran the following query against the database and found ten hosts with way more than 200 entries:

            select hostname, count (*) as entries from rmd_sensorevents group by hostname order by entries desc

            And furthermore, it seems like the problems began on the 12:th of march, seven of the hosts started spamming on different times that date, one on the 11:th and two yesterday...
            • 3. RE: Rogue System start/stop events overloading system
              Hello scorchspam,

              other than an epo-agent, systems with rsd-sensors need to be able to resolve the epo servername to its ip-address. so it can happen that the epo-agent on one of these systems can connect to the epo-server and upload its events, but the sensor not.
              i have never seen that a sensor restarted, when it was not able to connect, but maybe you could check that?