Incidents page fails to load due to huge number of alerts waiting to be processed in RSA Security Analytics 10.5

Issue

Incidents page on the User Interface (UI) shows as blank.

Even though rsa-im service is running, the Administration -> Services Page shows it as down

src=https://netwitness.file.force.com/servlet/rtaImage?eid=ka04z000000kMTM&feoid=00N5Y00000TPmR9&refid=0EM4z0000052xkl

Cause

This can happen when the Incident Management (IM) database grows to an extend that it crashes the IM service.

The database size may be huge because of the number of alerts that are coming in are too high.

Resolution

Stop the Incident Management service with the following command:

 # service rsa-im stop

Check the current size of the IM databse with the following command:

 # mongo im -u im -p im
 
 > show collections
 
system.indexes 3.48KB (uncompressed), 32.00KB (compressed)
 
system.users NaNundefined (uncompressed), NaNundefined (compressed)
 
categories 16.61KB (uncompressed), 32.00KB (compressed)
 
aggregation_rule 14.05KB (uncompressed), 32.00KB (compressed)
 
alert 1.17TB (uncompressed), 123.91GB (compressed)
 
incident 509.28KB (uncompressed), 1.16MB (compressed)
 
remediation_task 0.00B (uncompressed), 48.50KB (compressed)
 
tracking_id_sequence 182.00B (uncompressed), 32.00KB (compressed)
 
fs.files 0.00B (uncompressed), 48.50KB (compressed)
 
fs.chunks 0.00B (uncompressed), 48.50KB (compressed)

Remove old alerts from database with the following command by changing the date accordingly:

 > db.alert.remove({receivedTime : { $gte: ISODate('2016-01-01'), $lt: ISODate('2016-02-02')}})

Once removed, re-index the database to free up space:

 > db.alert.reIndex()

Once the database is cleared, check the rabbitmq queue for im.alert_queue.
Access the RabbitMQ web interface following the instructions in the article entitled How to access the RabbitMQ web interface to monitor and troubleshoot the event broker on an RSA Security Analytics Log Collector.

In the Web Interface, click on the im.alert.queue.

src=https://netwitness.file.force.com/servlet/rtaImage?eid=ka04z000000kMTM&feoid=00N5Y00000TPmRL&refid=0EM4z0000052xko

Check the number of queues messages.
If the number is above 100,000 , the IM service may crash again after the restart.
To purge the queue, scroll down and click on the "Purge" button.

NOTE: Any purged messages cannot be retrieved

Once the queue is purged, start the IM service on the SA head using the following command:

 # service rsa-im start

Notes

The RabbitMQ web interface should be opened from the SA server.

Product Details

RSA Product Set: RSA Security Analytics
RSA Version/Condition: 10.5.x

Approval Reviewer Queue

ASOC Approval Group