Skip to content
  • There are no suggestions because the search field is empty.

NetWitness Platform Host is dropping internal system messages due to rate-limiting messages

Issue

NetWitness hosts are dropping system messages due to rate-limiting.with 
errors similar to the following are reported  in the /var/log/messages file: 
rsyslogd: imuxsock begins to drop messages from pid "123"  due to rate-limiting

Cause

Rate-limiting set to a lower value than the current rate under rsyslog config settings.


Resolution

Please note the messages being dropped are the rsyslog system messages and not related to the logs being collected by any other event source.

To fix the below error, please check the rate limit interval and burst settings under rsyslogd.conf file.  This file can be found in /etc/rsyslog.conf.

Jun 2 13:47:38 rsa rsyslogd-2177: imuxsock lost 1504 messages from pid 7651 due to rate-limiting 
Jun 2 13:47:38 rsa rsyslogd-2177: imuxsock begins to drop messages from pid 7651 due to rate-limiting 
Jun 2 13:47:44 rsa  rsyslogd-2177: imuxsock lost 1433 messages from pid 7651 due to rate-limiting"

If the values for the below parameters in the config file are not set or set to a lower value, update the config file with the entries below:

$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 200

This implies rate limiting will be applied if more than 200 messages are received in an interval of 5 secs.

An example of doing this:

[root@NEW-NW11-NW-NODE-ZERO ~]# cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
[root@NEW-NW11-NW-NODE-ZERO ~]# vi /etc/rsyslog.conf

Add this entry to the bottom if it's not already present:
 

### Rate Limiting Rule ####
$ModLoad imuxsock # ( provides support for local system logging )
$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 200
### End of Rate Limiting Rule ###

Reload and confirm it loads properly:
 

[root@NEW-NW11-NW-NODE-ZERO ~]# systemctl daemon-reload
[root@NEW-NW11-NW-NODE-ZERO ~]# systemctl restart rsyslog
[root@NEW-NW11-NW-NODE-ZERO ~]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2024-04-30 19:20:58 UTC; 3s ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 22779 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           ├─22779 /usr/sbin/rsyslogd -n
           └─22784 /usr/sbin/rsa_audit_onramp --node_id=c685bd7a-b253-4514-ba9a-898fcd819bf1

Apr 30 19:20:58 NEW-NW11-NW-NODE-ZERO systemd[1]: Starting System Logging Service...
Apr 30 19:20:58 NEW-NW11-NW-NODE-ZERO rsyslogd[22779]:  [origin software="rsyslogd" swVersion="8.24.0-57.el7_9.3" x-pid="22779" x-info="http://www.rsyslog.com"] start
Apr 30 19:20:58 NEW-NW11-NW-NODE-ZERO systemd[1]: Started System Logging Service.
Apr 30 19:20:59 NEW-NW11-NW-NODE-ZERO rsa_audit_onramp[22784]: Loaded message bus configuration from /etc/carlos/carlos-rmq.config

Once applied, please monitor /var/log/messages to ensure we no longer see this error.


Internal Comments

UserName:saxonj
7/10/2014 8:15:18 PM - Technical Errors Noted
Tom Fedorchuk found some techincal errors in the article. I verified and notified the author via email and marked the article "Rejected for Tech Review" Johri, Earlier today, Tom Fedorchuk was referring to Primus Article a66583 and we had some questions about the article. In the "Fix� statement you refer to a file named "rsyslogd.conf� but you do not indicated where the file is stored. On our test lab servers the file is /etc/rsyslog.conf. We assume you intended that file. Also, the command to restart rsyslog is not correct. You indicated that the command was "/etc/init.d/rsyslogd restart�. It appears the correct command is "/etc/init.d/rsyslog restart�. Can you review the solution and make those changes? I have temporarily marked it as submitted for review until it is corrected. Fix Please note the messages being dropped are the rsyslog system messages and not related to the logs being collected by any other event source. To fix the below error, please check the rate limit interval and burst settings under rsyslogd.conf file. Jun 2 13:47:38 rsa rsyslogd-2177: imuxsock lost 1504 messages from pid 7651 due to rate-limiting Jun 2 13:47:38 rsa rsyslogd-2177: imuxsock begins to drop messages from pid 7651 due to rate-limiting Jun 2 13:47:44 rsa rsyslogd-2177: imuxsock lost 1433 messages from pid 7651 due to rate-limiting" If the values for the below parameters in the config file is not set or set to a lower value, update the config file with the entries below. $ModLoad imuxsock # ( provides support for local system logging ) $SystemLogRateLimitInterval 5 $SystemLogRateLimitBurst 200 This implies rate limiting will be applied if more than 200 messages are received in an interval of 5 secs. Please restart the rsyslogd service for the changes to take effect. #/etc/init.d/rsyslogd restart Once applied, please monitor /var/log/messages to ensure we no longer see this error. ----- Jonathan Saxon Technical Support Engineer RSA The Security Division of EMC2 My business hours are Monday-Friday 08:00am-17:00 EST (UTC/GMT+5), excluding American holidays.

UserName:jmarcinkowski
8/7/2014 12:02:47 PM - Changes have been made.
Changes have been made.

UserName:shurtj
8/11/2014 3:04:14 PM - Updated Article
Updated article and made changes to abide by Primus best practices. Changed audience to internal.

Evan Pols -- 30 Apr 2024
Updated title, rewrote with latest syntax and examples, adjusted applies to and cleaned up formatting with code blocks.

Product Details

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: All Nodes
NetWitness Version/Condition: 11.x, 12.x
Platform: CentOS 7, AlmaLinux


Approval Reviewer Queue

Technical approval queue