Skip to content
  • There are no suggestions because the search field is empty.

/var/log 10GB filesystem full due to rabbitmq logs on NetWitness 11.6-11.7.x

Issue

Rabbitmq can fill up the 10G /var/log/ partition (shared with all other logs) and cause component services to fail or fail to start after a reboot. Logrotate is a service that reviews all NW logs every hour, compresses recent logs and deletes the oldest of the logs, however it may not be aggressive enough or misses old logs with a different naming convention from previous versions.


Cause

This can happen for one of two reasons:
  1. The logrotate conf file only looks for compressed rabbitmq log files to delete, but older versions of NW (pre 11.6) did not compress the older logs, so when an upgraded system's logrotate runs, it doesn't match on the older ( uncompressed) logs and they remain on the filesystem. These logs are large and can contribute to the lack of space on /var/log/
  2. The logrotate conf file is not aggressive enough on busy systems and may need to be altered to keep fewer and smaller historic copies.

Workaround

If the root cause is that recent logs are simply too large and we're keeping too many copies, a NW administrator can edit the logrotate settings with the following steps:
  1. Edit the following file with vi:
    1. vi /etc/logrotate.d/rabbitmq-server
  2. Adjust the following two lines
    1. size 1G
      1. This tells logrotate to wait until a log file is 1 GB before compressing and rolling it out. This value can be adjusted down to a  lower value of 256M or 512M 
    2. rotate 4
      1. This tells logrotate to keep 4 historic copies of the compressed log before deleting it. This value can be adjusted down to 2 or 1 if necessary. 
  3. Test a manual log rotate run with debug mode to ensure the syntax is still working and get an idea of what it would do on the next run:
    1. /usr/sbin/logrotate -d /var/lib/logrotate.status /etc/logrotate.conf
  4. (Optional) - Run the logrotate manually instead of waiting for the cronjob:
    1. /usr/sbin/logrotate -s  /var/lib/logrotate/logrotate.status /etc/logrotate.conf

Resolution

For the first issue, the NW administrator can simply go in and delete the uncompressed logs under /var/log/rabbitmq. These log's names would resemble the following format:

           rabbit@7ccc3524-5e03-48ad-9229-e6bbaa84dd37.log-2021092016

Note the datetime appended to the end of the log name which shows this is a log from 2021 09-20-16. It is also not compressed, as compared to a newly rotated log post 11.6+

           rabbit@7ccc3524-5e03-48ad-9229-e6bbaa84dd37.log-2023010916.gz

Notes

  • Reducing the size and historic copies of logs may affect NW Supports capability to diagnose problems in the future. 
  • These steps DO NOT APPLY to 12.x
Example of logrotate being run in debug mode  
 
[root@NW11-LOG-HYBRID rabbitmq]# /usr/sbin/logrotate -d /var/lib/logrotate.status /etc/logrotate.conf
error: cannot stat /var/lib/logrotate.status: No such file or directory
reading config file /etc/logrotate.conf
including /etc/logrotate.d
Ignoring rabbitmq-server.rpmnew, because of .rpmnew ending
Ignoring .rabbitmq-server.swp, because of .swp ending
reading config file bootlog
reading config file firewalld
reading config file nw-service-topology
reading config file rabbitmq-server
reading config file salt
reading config file syslog
reading config file vsftpd
reading config file yum
Allocating hash table for state file, size 15360 B

Handling 13 logs

rotating pattern: /var/log/boot.log
after 1 days (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/boot.log
log does not need rotating (log is empty)
rotating pattern: /var/log/firewalld weekly (4 rotations)
empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed
considering log /var/log/firewalld
log /var/log/firewalld does not exist -- skipping

rotating pattern: /var/log/netwitness/nw-service-topology/*.log 104857600 bytes (2 rotations)
empty log files are rotated, old logs are removed
No logs found. Rotation not needed.

rotating pattern: /var/log/rabbitmq/*.log 1048576 bytes (2 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/rabbitmq/rabbit@7ccc3524-5e03-48ad-9229-e6bbaa84dd37.log
log does not need rotating (log size is below the 'size' threshold)
considering log /var/log/rabbitmq/rabbit@7ccc3524-5e03-48ad-9229-e6bbaa84dd37_upgrade.log
log does not need rotating (log size is below the 'size' threshold)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/salt/master weekly (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/salt/master
log does not need rotating (log is empty)
rotating pattern: /var/log/salt/minion weekly (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/salt/minion
log does not need rotating (log has been rotated at 2023-1-8 0:1, that is not week ago yet)

rotating pattern: /var/log/salt/key weekly (7 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/salt/key
log /var/log/salt/key does not exist -- skipping

rotating pattern: /var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
262144000 bytes (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/cron
log does not need rotating (log size is below the 'size' threshold)
considering log /var/log/maillog
log does not need rotating (log size is below the 'size' threshold)
considering log /var/log/messages
log does not need rotating (log size is below the 'size' threshold)
considering log /var/log/secure
log does not need rotating (log size is below the 'size' threshold)
considering log /var/log/spooler
log does not need rotating (log size is below the 'size' threshold)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/vsftpd.log weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/vsftpd.log
log /var/log/vsftpd.log does not exist -- skipping

rotating pattern: /var/log/xferlog weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/xferlog
log /var/log/xferlog does not exist -- skipping

rotating pattern: /var/log/yum.log yearly (4 rotations)
empty log files are not rotated, log files >= 30720 are rotated earlier, old logs are removed
considering log /var/log/yum.log
log does not need rotating (log has been rotated at 2023-1-1 0:1, that is not year ago yet)

rotating pattern: /var/log/wtmp monthly (1 rotations)
empty log files are rotated, only log files >= 1048576 bytes are rotated, old logs are removed
considering log /var/log/wtmp
log does not need rotating ('misinze' directive is used and the log size is smaller than the minsize value
rotating pattern: /var/log/btmp monthly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/btmp
log does not need rotating (log has been rotated at 2023-1-1 0:1, that is not month ago yet)


Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Local Log Collector/Log Decoder/Log Hybrid
RSA Version/Condition: 11.6.x, 11.7.x
Platform: CentOS
O/S Version: 7

Summary

Large volumes of rabbitmq logs could fill up the 10G /var/log partition causing rabbitmq and potentially other services to fail to start.


Approval Reviewer Queue

Technical approval queue