Reprocessing Orphaned RDQ files for RSA Netwitness Log Decoders/Log Collectors

Issue

Under very specific circumstances it may be required to remove some or all RDQ files from the RabbitMQ's Mnesia directory. When this is done, the RDQ files can no longer be processed directly by RabbitMQ due to inconsistencies within the internal data stores that moving these files causes.

Resolution

Important Note: Do not intentionally move RDQ files from the Mnesia directory unless you know what you are doing and have a good reason for doing it. Consult a senior engineer if necessary.

How To Reprocess RDQ Files

The process of re-injecting (reprocessing) RDQ files into RabbitMQ uses two tools: NwAMQPReceiver and NwAMQPSender

The idea is that the NwAMQPReceiver will read the messages from a directory of RDQ files and write them out in the format that RabbitMQ would normally use when processing log messages. Once these RDQ files are converted to these NGCE files, the NwAMQPSender is used to replay the messages back into RabbitMQ for processing.

Note: Converting RDQ files into NGCE files uncompresses the RDQ file. Make sure that there is enough free space on the partition being used to process these files.Try a small sample of files first to confirm enough space is available and the amount of time required to fully process all files.

1. Install the nwlogcollectortools rpm package to gain access to the NwAMQPReceiver and NwAMQPSender

This package is not in the standard repository and may not already be installed. It is important to get the version that matches the log collector version. See the internal notes below. Run rpm -qa | grep -i nwlogcollectortools and confirm the package is installed.

2. Move the RDQ files to the Log Collector that will be responsible for file processing. This is to provide for better performance being on the same server as the Log Collector service.

3. Confirm that Screen is installed as processing RDQ files can take a long time. This will allow the receiver and sender to run even when logged out.
- rpm -qa | grep screen
- If it is not installed run yum install screen

4. Start Screen by typing screen at the command prompt. Run man screen to review the Screen program options if needed.

5. Create a temporary holding location for the RDQ files
Ex. mkdir /tmp/rdqfiles

6. Create a temporary holding location for the NGCE files (Remeber uncompressed RDQ files can be very large. Make sure there is plenty of space available.)
Ex. mkdir /tmp/ngcefiles

7. If planning on using verbose mode on the sender and receiver it is important to know that there can be a lot of output per file. To capture this output to a file instead of having it output to the screen, add the following to the end of the NwAMQPReceiver and NwAMQPSender commands:

Example: NwAMQPReceiver 1> /$(date +%F).log
Example Usage: NwAMQPReceiver --fromdir /tmp/rdq --dir /tmp/ngce 1> /tmp/RDQLogging/$(date+%F).log
Note: It is suggested not to leave verbose logging enabled if processing a large number of RDQ/NGCE files as the log file will be very large

8. Run the NwAMQPReceiver on the RDQ files.
Example Usage:. NwAMQPReceiver --fromdir --dir

The receiver needs only the source directory of RDQ files and the target directory of where the NGCE files will be created.
A sample NGCE file will look like this: 654_rdq-161478571600006241.ngce
The file prefix is the name of the RDQ file that it was extracted from (654_rdq), then a unix time stamp followed by a sequence of numbers that represent the index of the first message in the RDQ file.
Receiver Options
--fromdir - [Manditory] Source directory that contains the RDQ files to be processed
--dir - [Manditory] Destination directory that will contain the NGCE files
--delete-rdq-after=0 - This option deletes the RDQ file after it has been successfully processed 0 = Do Not Delete [Default], 1 = Delete
--verbose true - Produces output of each message file as it is being processed from the RDQ file to the screen
--echo-raw true - This option displays all meta found within each event in the message file, useful for debugging actual messages. Causes processing to take 60% longer.

9. Once the messages are extracted, run the NwAMQPSender on the new NGCE files. If there is an error during processing the NwAMQPSender will stop processing and will need to be restated.
Example Usage: NwAMQPSender --dir --exchange file --routing-key file --num-messages=0 --maxeps 1000

Sender Options
--dir - [Manditory] Source directory that contains the NGCE files to be processed
--excahnge - [Manditory] Set to file when the sender is being ran on a CentOS Collector. Set to windowslegacy when ran on a Windows Collector.
--routing-key - [Manditory] Set to file when the sender is being ran on a CentOS Collector. Set to windowslegacy when ran on a Windows Collector.
--num-messages=0 - [Manditory] When set to 0 the Sender will process all NGCE files in the processing directory
--maxeps 1000 - Allows setting the maximum events per second to inject. The sender will try to enforce the specific limit based on current load. Ex: 1000 = 1K eps
--delete-files 1 - This tells the sender that the successfully processed NGCE file should be deleted after processing. 0 = Do Not Delete, 1 = Delete [Default]
--verbose true - Produces output that shows what NGCE file is being processed and how many messages were published from the file

10. Once all the files are processed all the log messages are now within the system and ready for investigations.

Product Details

RSA Product Set: Netwitness Packet & Logs
RSA Product/Service Type: Log Decoder
RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x

Summary

Under some circumstances Rabbitmq RDQ files can become orphaned from Log Decoder processing. When that happens this RDQ files will require manual processing so log messages are not lost.

Approval Reviewer Queue

ASOC Approval Group