RSA NetWitness LogCollection is not working due to rabbitmq vhost down

Issue

While Collector and Rabbitmq services are running fine, the log collection is not working with the below errors.

 /var/log/messages:
 
Sep 8 08:08:24 NWCollector NwLogCollector[2360]: [BufferedChannel] [failure] An error occurred publishing to an AMQP channel: connection error: 541: AMQP_CONNECTION_OPEN_METHOD caused: INTERNAL_ERROR - access to vhost 'logcollection' refused for user 'logcollector': vhost 'logcollection' is down
 
Sep 8 08:08:24 NWCollector NwLogCollector[2360]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: connection error: 541: AMQP_CONNECTION_OPEN_METHOD caused: INTERNAL_ERROR - access to vhost 'logcollection' refused for user 'logcollector': vhost 'logcollection' is down
 
Sep 8 08:08:24 NWCollector NwLogCollector[2360]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: connection error: 541: AMQP_CONNECTION_OPEN_METHOD caused: INTERNAL_ERROR - access to vhost 'logcollection' refused for user 'logcollector': vhost 'logcollection' is down

 /var/log/rabbitmq/<nodeid>.log:
 
2020-09-08 08:47:24.912 [error] <0.27103.2> Error on AMQP connection <0.27103.2> (127.0.0.1:53344 -> 127.0.0.1:5671, vhost: 'none', user: 'logcollector', state: opening), channel 0:
 
 {handshake_error,opening,
 
 {amqp_error,internal_error,
 
 "access to vhost 'logcollection' refused for user 'logcollector': vhost 'logcollection' is down",
 
 'connection.open'}}

Cause

The issue occurs when rabbitmq vhosts corrupted due to obrupt shutdown of system.

Resolution

Please follow the below steps to fix the issue.

Stop rabbitmq-server service using systemctl stop rabbitmq-server.service.
Move contents of /var/netwitness/rabbitmq/mnesia/rabbit@ /msg_stores/vhosts directory to backup location using below commands.

cd /var/netwitness/rabbitmq/mnesia/rabbit@ /msg_stores/vhosts
mv * /root/oldrabbitmqvhosts

Start rabbitmq-server service using systemctl start rabbitmq-server.service.
Verify contents regenerated for /var/netwitness/rabbitmq/mnesia/rabbit@ /msg_stores/vhosts directory.
Verify the Investigate page to see the latest logs coming from Collector.

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.4.1.2
Platform: CentOS
O/S Version: 7

Summary

This document outlines the procedure to fix rabbitmq vhost issue and to start collection working.

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue