I thought I'd post this here in case others run across the same issue - had not seen anything in the community forums or the release notes about it.
We recently upgraded our Web Gateways to version 22.214.171.124 code (previously running 7.6.2) and we noticed in our SIEM that the indexing had a major drop in volume from the Web Gateways. We log to our SIEM using syslog, so we pointed a new syslog feed to a test SIEM to see if there was a configuration problem on the SIEM's incoming traffic. We saw the same results - a significant lower volume of logs than expected. Loading a website would generate 70-80 lines of logs in the access.log files, but only 4-5 line items in the SIEM.
We checked the rulesets and rsyslog.conf file, but could not find any changes that would account for the issue. However, we did find that the event count in the SIEM was pegged at 200 events every 5 seconds, so we assumed there must be some throttling or limit on the Web Gateway outgoing syslog, rather than just randomly dropped logs.
Ran the following command after SSH into the Web Gateway...
tail - f /var/log/messages
... and immediately founds logs every five seconds stating:
imuxsock lost 2165 messages from pid #### due to rate-limiting (the value in red was different in each log line)
imuxsock begins to drop messages from pid #### due to rate-limiting
We found the following documentation for RSYSLOG that describes the imuxsock module and rate-limiting, and what's the default rate-limit, if not specified? 200 messages per 5 seconds; exactly what we were getting in the SIEM.
So we went into our rsyslog.conf file under Configuration -> File Editor -> Appliance Name -> rsyslog.conf
Under the following line, we added the lines below in blue:
Immediately, we saw the jump in log traffic to the SIEM and are no longer seeing the rate-limiting message in the log files. We have not seen any performance hit in CPU or RAM since the change either, and the Web Gateways were logging roughly 3000 messages per 5 seconds before the upgrade without any issue, so we don't expect any CPU or RAM issues.
Thanks for the information. We were also having this issue. I wanted to also mention that for our environment, we were processing a considerably larger amount of messages and the above settings did not work for us. Of course it is on an individual basis, but I wanted to note that another option, if you want to let SYSLOG roam free, is to use the following settings:
We also have not noticed any CPU or Memory issues with these.