Skip to content
  • There are no suggestions because the search field is empty.

RSA NetWitness Platform - Error 'disk resource limit alarm has tripped' in sa@localhost.log due to file collection logs piled up in Collector

Issue

Rabbitmq server is not running due to ' disk resource limit alarm' as below.

/var/log/rabbitmq/sa@localhost.log:
=WARNING REPORT==== 29-Aug-2019::09:26:47 ===
disk resource limit alarm has tripped on node sa@localhost. Collection will be blocked until this alarm clears!

/var/log/messages:
Aug 24 04:05:12 Collector1 NwLogCollector[23993]: [MessageBrokerLogReceiver] [info] info 2019-08-24T04.05.12Z Disk free space insufficient. Free bytes:104907436032 Limit:104908750000

Cause

This issue is due to /var/netwitness/logcollector has reached 80% of available storage as below.

[root@Collector1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root
20G 4.5G 14G 25% /
tmpfs 48G 0 48G 0% /dev/shm
/dev/sdd1 248M 150M 85M 64% /boot
/dev/mapper/VolGroup00-usrhome
3.9G 417M 3.3G 12% /home
/dev/mapper/VolGroup02-tmp
20G 171M 19G 1% /tmp
/dev/mapper/VolGroup02-varlog
9.8G 3.6G 5.8G 39% /var/log
/dev/mapper/VolGroup01-nwhome
10G 935M 9.1G 10% /var/netwitness
/dev/mapper/VolGroup02-concroot
30G 940M 30G 4% /var/netwitness/concentrator
/dev/mapper/VolGroup03-concinde
300G 38G 263G 13% /var/netwitness/concentrator/index
/dev/mapper/VolGroup02-concmeta
2.4T 2.3T 130G 95% /var/netwitness/concentrator/metadb
/dev/mapper/VolGroup02-concsess
300G 285G 16G 95% /var/netwitness/concentrator/sessiondb
/dev/mapper/VolGroup01-lcol
489G 362G 127G 80% /var/netwitness/logcollector
/dev/mapper/VolGroup01-ldecroot
30G 923M 30G 4% /var/netwitness/logdecoder
/dev/mapper/VolGroup01-ldecinde
10G 37M 10G 1% /var/netwitness/logdecoder/index
/dev/mapper/VolGroup01-ldecmeta
300G 284G 17G 95% /var/netwitness/logdecoder/metadb
/dev/mapper/VolGroup01-ldecpack
2.8T 2.7T 149G 95% /var/netwitness/logdecoder/packetdb
/dev/mapper/VolGroup01-ldecsess
30G 29G 2.0G 94% /var/netwitness/logdecoder/sessiondb
/dev/mapper/VolGroup03-warec
400G 35G 366G 9% /var/netwitness/warehouseconnector
/dev/mapper/VolGroup00-vartmp
5.8G 12M 5.5G 1% /var/tmp
[root@hydsiemhyb01 ~]#

Resolution

Please use the below steps to identify the cause for high consumption in /var/netwitness/logcollector.
  1. Run du -xh /var/netwitness/logcollector --max-depth=2|sort -h>collectingspace.txt command for consumption details.
  2. tail collectingspace.txt command shows as below.
    20M /var/netwitness/logcollector/rabbitmq/log
    27M /var/netwitness/logcollector/rabbitmq
    791M /var/netwitness/logcollector/statdb
    3.3G /var/netwitness/logcollector/metadb
    17G /var/netwitness/logcollector/upload/microsoft_dhcp_2008
    373G /var/netwitness/logcollector/upload/iis_tvm
    389G /var/netwitness/logcollector/upload
    389G /var/netwitness/logcollector/upload_chroot
    389G /var/netwitness/logcollector/upload_chroot/home
    782G /var/netwitness/logcollector/
  3. Above output indicates that the space consumption was due to /var/netwitness/logcollector/upload/iis_tvm which has huge files under /var/netwitness/logcollector/upload/iis_tvm/ /save directory.
  4. These files were saved after a successful process of logs by Log Collector by choosing "Save on Success" as below for File Collection event source.
    Save
  5. Remove the files under /var/netwitness/logcollector/upload/iis_tvm/ /save directory to free up space.
  6. service rabbitmq-server start command in collector.
  7. Uncheck the "Save On Success" option for all file collection event sources in Collector->Config->Event Sources->File/Config page.
More details on Save On Success

Product Details

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Collector
RSA Version/Condition: 10.6.X, 11.X
Platform: CentOS
O/S Version: 7

Summary

This document outlines the procedure to fix the rabbitmq issues when /var/netwitness/logcollector reached 80% full due to file collection logs.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue