Connectivity to RSA Security Analytics Health and Wellness and other services may appear down in the GUI when they are not.

Issue

Connectivity to Health and Wellness and other services may appear down in the GUI when they are not. Rabbitmq is either crashing or running sluggish, and there are a large number of rdq files backed up in the rabbitmq queue.

Cause

Various services are configured to send statistics far too often, which can overwhelm the rabbitmq messaging bus. This can also affect the overall connectivity of Security Analytics and other services. This is typically a misconfiguration of check intervals being sent to the Security Analytics server.

Workaround

Increase the intervals at which the services are configured on SA server to send the statistics.

To check whether your system is affected by this, you can run this command on the Security Analytics server appliance:

 [root@rsaaio ~]# 
 updatedb && locate --regex ".conf.erb$" | xargs grep "interval"
 
/etc/puppet/modules/archiver/templates/NwArchiver.conf.erb: interval "10"
 
/etc/puppet/modules/archiver/templates/NwArchiver.conf.erb: interval "60"
 
/etc/puppet/modules/broker/templates/NwBroker.conf.erb: interval "5"
 
/etc/puppet/modules/broker/templates/NwBroker.conf.erb: interval "60"
 
/etc/puppet/modules/concentrator/templates/NwConcentrator.conf.erb: interval "10"
 
/etc/puppet/modules/concentrator/templates/NwConcentrator.conf.erb: interval "60"
 
/etc/puppet/modules/decoder/templates/NwDecoder.conf.erb: interval "10"
 
/etc/puppet/modules/decoder/templates/NwDecoder.conf.erb: interval "60"
 
/etc/puppet/modules/ipdbextractor/templates/NwIPDBExtractor.conf.erb: interval "10"
 
/etc/puppet/modules/ipdbextractor/templates/NwIPDBExtractor.conf.erb: interval "60"
 
/etc/puppet/modules/logcollector/templates/NwLogCollector.conf.erb: interval "10"
 
/etc/puppet/modules/logcollector/templates/NwLogCollector.conf.erb: interval "60"
 
/etc/puppet/modules/logdecoder/templates/NwLogDecoder.conf.erb: interval "10"
 
/etc/puppet/modules/logdecoder/templates/NwLogDecoder.conf.erb: interval "60"
 
/etc/puppet/modules/logdecoder/templates/NwLogDecoder_ESM.conf.erb: interval "600"
 
/etc/puppet/modules/logdecoder/templates/NwLogDecoder_ESM.conf.erb: interval "60"
 
/etc/puppet/modules/warehouseconnector/templates/NwWarehouseConnector.conf.erb: interval "10"
 
/etc/puppet/modules/warehouseconnector/templates/NwWarehouseConnector.conf.erb: interval "60"
 
/etc/puppet/modules/workbench/templates/NwWorkbench.conf.erb: interval "10"
 
/etc/puppet/modules/workbench/templates/NwWorkbench.conf.erb:# interval "60"

The correct intervals should be 60 and 180 seconds respectively.

The one line workaround that should also be run on the SA Server appliance is this:

 updatedb && for files in $(locate --regex ".conf.erb$");do sed -i 's/interval "60"/interval "180"/g' $files; sed -i 's/interval "10"/interval "60"/g' $files;done && sed -i 's/interval 60/interval 180/g' /etc/puppet/modules/rsa-sms-server/files/_collectd_java.conf && sed -i 's/interval "5"/interval "60"/g' /etc/puppet/modules/broker/templates/NwBroker.conf.erb && service puppet restart

Product Details

RSA Product Set: Security Analytics
RSA Version/Condition: 10.4.1,10.4.1.2, 10.4.1.3, 10.5.x

Summary

This article describes troubleshooting steps when connectivity to Health and Wellness and other services may appear to be down when they are actually not.

Approval Reviewer Queue

ASOC Approval Group