Skip to content
  • There are no suggestions because the search field is empty.

How to reinstall rsa-sms and rabbitmq in NetWitness Platform 11.3.x node zero

Issue

Sometimes NetWitness UI can be slow down due to unknown SMS failures coupled with rabbitmq failures.
[scheduler_Worker-5] INFO com.rsa.netwitness.sms.core.persistence.RRDTool - Data fetched. 62 points obtained
[scheduler_Worker-4] INFO com.rsa.smc.esm.core.alert.BaselineAggregationCleanupService - removed 0 old documents removed from collection
[scheduler_Worker-1] INFO com.rsa.smc.esm.core.jobs.DiscoveryPersistenceJob - started
[scheduler_Worker-3] INFO com.rsa.smc.esm.core.jobs.DiscoveryPersistenceJob - started
[scheduler_Worker-4] INFO com.rsa.smc.esm.core.jobs.DiscoveryPersistenceJob - started
[scheduler_Worker-1] INFO com.rsa.smc.esm.core.jobs.DiscoveryPersistenceJob - started
[scheduler_Worker-1] ERROR com.rsa.smc.esm.core.jobs.NotificationDispatchJob - Error occurred in notification dispatch job: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=379ae2db-eba4-490e-a319-9739786e4fde:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.SocketTimeoutException: connect timed out}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=379ae2db-eba4-490e-a319-9739786e4fde:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.SocketTimeoutException: connect timed out}}]
[scheduler_Worker-2] ERROR com.rsa.smc.esm.core.jobs.LogDecoderMappingsJob - Error occurred in log decoder mapping dispatch job: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=379ae2db-eba4-490e-a319-9739786e4fde:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.SocketTimeoutException: connect timed out}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=379ae2db-eba4-490e-a319-9739786e4fde:27017, type=Unknown, state=Connecting, exception={com.mongodb.MongoException$Network: Exception opening the socket}, caused by {java.net.SocketTimeoutException: connect timed out}}]
[AMQP Connection 127.0.0.1:5671] ERROR com.rabbitmq.client.impl.ForgivingExceptionHandler - An unexpected connection driver error occurred
com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds
at com.rabbitmq.client.impl.AMQConnection.handleSocketTimeout(AMQConnection.java:784)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:684)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:603)
at java.lang.Thread.run(Thread.java:748)
And you may see rsa-sms is not recovering and going into a failed state continuously.
# service rsa-sms status
Redirecting to /bin/systemctl status rsa-sms.service
??rsa-sms.service - RSA NetWitness SMS :: Server
Loaded: loaded (/usr/lib/systemd/system/rsa-sms.service; enabled; vendor pres
Active: failed (Result: exit-code) since Fri 2020-07-24 11:05:15 KST; 4h 27mi
Process: 54273 ExecStart=/opt/rsa/sms/bin/sms start (code=exited, status=0/SUC
Main PID: 54327 (code=exited, status=1/FAILURE)Jul 24 10:59:18 SEC-FRSH-ICB301 systemd[1]: Starting RSA NetWitness SMS :: Serve
Jul 24 10:59:18 xxxxxx sms[54273]: Starting RSA NetWitness SMS :: Serve
Jul 24 10:59:18 xxxxxx systemd[1]: Started RSA NetWitness SMS :: Server
Jul 24 11:05:15 xxxxxx systemd[1]: rsa-sms.service: main process exited
Jul 24 11:05:15 xxxxxx systemd[1]: Unit rsa-sms.service entered failed
Jul 24 11:05:15 xxxxxx systemd[1]: rsa-sms.service failed.


Resolution

In this case, you can try re-installing rsa-sms and rabbitmq to solve this issue.
Note) Those steps are validated in 11.3.x. If you want to apply them in 11.4.x above, you need to validate with CE.
  • The steps for rsa-sms re-installation
  1. Stop sms service:
    # service rsa-sms stop
     
  2. Move logs for backup
    # mv /opt/rsa/sms/logs/* /root
     
  3. Remove rsa-sms service.
    # yum remove rsa-sms-server
    # service rabbitmq-server restart
    Wait for 3mins after this step.
     
  4. Remove config files
    # rm -rf /etc/netwitness/platform/legacy_mongo/rsa-sms-server.mongodb.properties
    # rm -rf /etc/netwitness/platform/legacy_rabbit/rsa-sms-server.rabbitmq.properties
    Please validate that these files do not exist once removed.
     
  5. Change few mongo entries
    Login to Mongo using deploy_admin credentials.
    # mongo admin -u deploy_admin -p 'password'
    > use admin
    > db.system.users.deleteOne({"_id" : "sms.sms"})
    > db.system.users.deleteOne({"_id" : "esm.esm"})
    > db.system.users.deleteOne({"_id" : "les.les"})
    > db.system.users.deleteOne({"_id" : "asg.asg"})
    > exit
  6. Re-install service
    > chef-client -c /var/lib/netwitness/config-management/client.rb -r "recipe[rsa-sms-server]"
     
  7. Check rsa-sms service status
    This step could take some time to show it is in running state.
 
  • The steps for rabbitmq-server re-installation

0. Backup of head node: Since these are risky steps, take a complete backup on the admin node using the NRT tool.

1. Stop rabbitmq-server
# service rabbitmq-server stop

2. backup this directory /etc/rabbitmq/ssl/ and then run
# rm -rf /etc/rabbitmq/ssl/

3. Run chef install of rabbitmq
# chef-client -r "recipe[nw-rabbitmq]" --config /var/lib/netwitness/config-management/client.rb --json-attributes /etc/netwitness/config-management/node.json

4. Wait for completion and check the status of rabbitmq after 2 mins
# service rabbitmq-server status
If the rabbit is not up and running, please restore the admin server with the NRT tool

5. If the above step is running, restart sms service.
# service rsa-sms restart

If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.


Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.3.1.1
Platform: CentOS
O/S Version: 7

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue