RabbitMQ certificates pointing to incorrect certificate files resulting in orphan queues between Local Log Collector and Log Decoder in RSA Security Analytics

Issue

The RabbitMQ config file points to incorrect certificates files, this can be noticed by any of the below symptoms:

Symptom 1:

Queues on the Local Log Collector are missing consumers this can be confirmed by running the below command:

SSH to the Log Decoder:

 rabbitmqctl list_queues -p logcollection name consumers | column -t

If there are no consumers, the output should look like the below, where 0 means that there are no consumers to the corresponding queues:

 [root@logdecoder /]# rabbitmqctl list_queues -p logcollection name consumers | column -t
 
Listing queues ...
 
LogDecoder.logdecoder.checkpoint 0
 
LogDecoder.logdecoder.cmdscript 0
 
LogDecoder.logdecoder.file 0
 
LogDecoder.logdecoder.netflow 0
 
LogDecoder.logdecoder.odbc 0
 
LogDecoder.logdecoder.sdee 0
 
LogDecoder.logdecoder.snmptrap 0
 
LogDecoder.logdecoder.syslog 0
 
LogDecoder.logdecoder.vmware 0
 
LogDecoder.logdecoder.windows 0
 
LogDecoder.logdecoder.windowslegacy 0
 
rabbitmq.log 0

Symptom 2:

Errors similar to the below are noticed in the RabbitMQ log file under :

 /var/log/rabbitmq/rabbit\@<uuid>.log

  =ERROR REPORT==== 31-Oct-2016::13:09:09 ===
  
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate
  
=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
  
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate
  
=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
  
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate
  
=ERROR REPORT==== 31-Oct-2016::13:09:11 ===
  
Error on AMQP connection <0.12631.0>:
  
{ssl_upgrade_error,{tls_alert,"bad certificate"}}
  
=ERROR REPORT==== 31-Oct-2016::13:09:11 ===
  
Error on AMQP connection <0.12635.0>:
  
{ssl_upgrade_error,{tls_alert,"bad certificate"}}

Symptom 3:

Errors similar to the below are noticed Log Collector Logs under:

  /var/log/messages

  Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
  
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.windowslegacy] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket
  
Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
  
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.windows] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket
  
Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
  
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.vmware] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket

Cause

RabbitMQ certificates are not pointing to the correct files.

The truststore.pem certificate located under:

 /etc/netwitness/ng/rabbitmq/ssl/

should be pointing to the truststore.pem certificate file under:

 /etc/rabbitmq/ssl

The cert.pem and privkey.pem certificates under:

 /etc/netwitness/ng/rabbitmq/ssl/keys

should be pointing to cert.pem and key.pem certificates respectively under:

 /etc/rabbitmq/ssl/server/

Resolution

SSH to the Log Decoder and run the below commands:

 cd /etc/netwitness/ng/rabbitmq/ssl/
 
ln -s /etc/rabbitmq/ssl/truststore.pem truststore.pem
 
cd keys/
 
ln -s /etc/rabbitmq/ssl/server/cert.pem cert.pem
 
ln -s /etc/rabbitmq/ssl/server/key.pem privkey.pem

The output should be something like this:

  [root@NW11-LOG-HYBRID ~]# ls -l /etc/netwitness/ng/rabbitmq/ssl/
  
total 0
  
drwxr-xr-x. 2 rabbitmq rabbitmq 41 Jul 23  2020 keys
  
lrwxrwxrwx. 1 rabbitmq rabbitmq 32 Jul 23  2020 truststore.pem -> /etc/rabbitmq/ssl/truststore.pem
  
[root@NW11-LOG-HYBRID ~]# cd keys
  
-bash: cd: keys: No such file or directory
  
[root@NW11-LOG-HYBRID ~]# ls -l /etc/netwitness/ng/rabbitmq/ssl/keys
  
total 0
  
lrwxrwxrwx. 1 rabbitmq rabbitmq 33 Jul 23  2020 cert.pem -> /etc/rabbitmq/ssl/server/cert.pem
  
lrwxrwxrwx. 1 rabbitmq rabbitmq 32 Jul 23  2020 privkey.pem -> /etc/rabbitmq/ssl/server/key.pem

Product Details

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Log Collector
NetWitness Version/Condition: 12.x
Platform: CentOS/Alma Linux

Summary

Queues on Log Collector are missing consumers, in other words Log Decoders are not consuming from Local Log Collector because the RabbitMQ config file is pointing to incorrect certificate files.

Approval Reviewer Queue

Technical approval queue