Error message 'Disk resource limit alarm has tripped on node' in NetWitness

Issue

The following symtoms are observed when this problem occurs:

1)  /var/log/messages, a message similar to that below is seen
    

Resolution

Use the attached procedures to recover from this situation.

https://bedfordjira.na.rsa.net/secure/attachment/251395/Check_for_LC_stale_queues.pdf

ATTENTION: Do NOT ATTEMPT TO MANUALLY COPY/MOVE/DELETE ANY .rdq files fromthe RabbitMQ Mnesia directory! This will create inconsistency of RabbitMQ’s internal datastore and RESULTS IN DATA LOSS!! 

If you did move the RDQ Files then follow these instructions but the process is likely to take many days for a large number of RDQ files.

How To Reprocess Old RDQ Files

The idea is that the NwAMQPReceiver will read the messages from a directory of rdq files and write them out in the same file format the old version did while draining queues. Then you use the new NwAMQPSender just like you would with the old version. These versions have better stats etc. One feature of the new NwAMQPSender is that it has rate limiting i.e. you can set it to re-inject at 10k EPS and just let it continuously process messages instead of swamping the
LC with a huge spike.
 
I always advise moving the rdq files that are stale to an LC and processing there due to better performance on an LC.
 
Since the examples in the doc are for the windows version use the following :

copy the files to the least busy LC (create the folders mentioned below checking that the amount of storage available there is at least 10x the total rdq sizes for the extraction, these paths are just suggestions they can be any path)
 
create a /tmp/rdqs directory
 
then create a /tmp/ngce directory
 
Copy the new tools to the LC (overwrite the old ones if they are already on there or run the tools from the folder you copied them too).
 
extract the rdq’s to messages:
 
NwAMQPReceiver --echo true --verbose true --fromdir /tmp/rdqs --dir /tmp/ngce --delete-rdq-after=1
 
Re-inject those messages (at some nominal rate say 10k)…
 
NwAMQPSender --verbose true --dir /tmp/ngce --exchange file --routing-key file --num-messages=0 --delete-files=1 --maxeps 5000
 
ATTENTION #2: Be careful when using the option --delete-files=1. This will delete the files even if there was an error in reinjecting them, so only use once you have confirmed the files are being processed successfully.
 
Note the use of :
--delete-rdq-after=1 on the receiver tool which will delete each rdq that was successfully processed
And
--delete-files=1 on the sender tool which will delete each message file after successful processing.

Product Details

RSA Product Set: Security Analytics, NetWitness
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.x

Approval Reviewer Queue

Technical approval queue