The messages log file grows to fill the /var/log volume, preventing services from starting on RSA NetWitness Host

Issue

The /var/log/messages log file grows to occupy all of the available space in the /var/log partition preventing services such as the nwlogcollector (and other services) from starting.

For the failure of logrotate for other logs refer to the following KBs:
#000030086 RabbitMQ in NetWitness 10.4.0.2 - The /var/log partition becomes full on an RSA Security Analytics Log Collector due to rabbitmq log files not rotating
#000037185 logstash in NetWitness 11.x - RSA NetWitness 11.x /var/log mount is full due to logstash directory

Cause

This issue is most often seen in RSA NetWitness Hybrid and All-In-One (AIO) appliances and Virtual Log Collectors (VLCs) due to the volume of entries that the nwlogcollector service writes to /var/log/messages.
VLCs often have a smaller /var/log volume (e.g. 3.9G) than physical appliances (e.g. 9.8G).

In order to detect the problem, log into the affected host using SSH and run the following commands. The outputs in the examples below were taken from a VLC As can be seen in the hi-lighted section in red above, /var/log volume has reached 100% utilization.

To locate which files and directories are occupying the most space. We have identified the issue now, /var/log/messages is causing /var/log to fill up quickly.


An alternative way of doing this would be to use the 'ls' command and sorting file size to examine the files in /var/log directory (Hint: Could add the -R switch as well to recurse into subdirectories, however, the -S switch only sorts files within each directory):
Note: If the utilisation of 'df -hP' and 'du -ahx' don't match then this is likely due to a failure of logrotate when writing to a new file. Run the following command to check for deleted but not released log files: To release the space being taken by the deleted file (but held by rsyslogd as it still has an open file handle), you will either need to reboot the OS or restart the syslog services

Workaround

BEFORE:
The current configuration of logrotate for syslog services in 10.6.x is as follows:
AFTER:
This file needs to be edited to the following:
In this way we are going to rotate /var/log/messages on a weekly basis (retaining 4 compressed logs) or when the file reaches the size of 250 MB (whichever comes first).
The dateext means that the date of rotate will be appended to the filename e.g. messages-20190212

Test that the configuration is correct by running logrotate manually using the following command:
If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and reference this article for further assistance.

Resolution

The logrotate service's configuration need to be adjusted by editing /etc/logrotate.d/syslog to allow the normal rotation of /var/log/messages.

Notes

If after applying the above steps logrotate is not working, then the syslog service may need to be restarted as shown below.
Note: Other non-standard packages installed on the host such as syslog-ng may also cause logrotate to fail due to additional file handles on /var/log/messages. RSA Support would recommend that these non-standard packages be removed. You may be able to find these processes using the following command:

Internal Comments

Lee McCotter -- 13 Mar 2019
Unfortunately KB version prior to today may not work if /var/log/messages manages to fill volume within a single day (despite KBs extensive application on customer sites)

Editing /etc/logrotate.d/syslog and restarting OS to the following doesn't avoid logrotate failing due to lack of space when creating new /var/log/messages:

Product Details

RSA Product Set: NetWitness Logs & Network/Security Analytics
RSA Product/Service Type: NetWitness Appliances (including Hybrid & All-in-One appliances), VLC hosts
RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: 6

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue