1 Reply Latest reply on Mar 15, 2017 12:13 PM by punnettr

    Syslog Rate-Limiting after upgrade to 7.7.1.1

    ednegron

      I thought I'd post this here in case others run across the same issue - had not seen anything in the community forums or the release notes about it.

       

      We recently upgraded our Web Gateways to version 7.7.1.1 code (previously running 7.6.2) and we noticed in our SIEM that the indexing had a major drop in volume from the Web Gateways. We log to our SIEM using syslog, so we pointed a new syslog feed to a test SIEM to see if there was a configuration problem on the SIEM's incoming traffic. We saw the same results - a significant lower volume of logs than expected. Loading a website would generate 70-80 lines of logs in the access.log files, but only 4-5 line items in the SIEM.

       

      We checked the rulesets and rsyslog.conf file, but could not find any changes that would account for the issue. However, we did find that the event count in the SIEM was pegged at 200 events every 5 seconds, so we assumed there must be some throttling or limit on the Web Gateway outgoing syslog, rather than just randomly dropped logs.

       

      Ran the following command after SSH into the Web Gateway...

      tail - f /var/log/messages

       

      ... and immediately founds logs every five seconds stating:

      imuxsock lost 2165 messages from pid #### due to rate-limiting   (the value in red was different in each log line)

      imuxsock begins to drop messages from pid #### due to rate-limiting

       

      We found the following documentation for RSYSLOG that describes the imuxsock module and rate-limiting, and what's the default rate-limit, if not specified? 200 messages per 5 seconds; exactly what we were getting in the SIEM.

      What is imuxsock?

      Changing the settings

       

      So we went into our rsyslog.conf file under Configuration -> File Editor -> Appliance Name -> rsyslog.conf

      Under the following line, we added the lines below in blue:

      $ModLoad imuxsock.so

      $SystemLogRateLimitInterval 5

      $SystemLogRateLimitBurst 4000

       

      Immediately, we saw the jump in log traffic to the SIEM and are no longer seeing the rate-limiting message in the log files. We have not seen any performance hit in CPU or RAM since the change either, and the Web Gateways were logging roughly 3000 messages per 5 seconds before the upgrade without any issue, so we don't expect any CPU or RAM issues.

       

        • 1. Re: Syslog Rate-Limiting after upgrade to 7.7.1.1
          punnettr

          Thanks for the information. We were also having this issue. I wanted to also mention that for our environment, we were processing a considerably larger amount of messages and the above settings did not work for us. Of course it is on an individual basis, but I wanted to note that another option, if you want to let SYSLOG roam free, is to use the following settings:

           

          $SystemLogRateLimitInterval 0

          $SystemLogRateLimitBurst 0

           

          We also have not noticed any CPU or Memory issues with these.