Skip to content
  • There are no suggestions because the search field is empty.

Respond Service unavailable after upgrade to RSA NetWitness Platform 11.3.x

Issue

Respond service is unavailable in NetWitness GUI after upgrade from 11.2.0 to 11.3.1.1 as shown below.
User-added

Cause

For some reason, there exists ordering issue with the aggregation rules inside the mongo collection(aggregation_rule) after upgrade.
You may see following error messages from respond log.
2020-02-24 05:21:03,766 [ main] WARN o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext|Exception encountered during context initialization - canceling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'referenceDataLoader': Invocation of init method failed; nested exception is org.springframework.dao.DuplicateKeyException: E11000 duplicate key error collection: respond-server.aggregation_rule index: name_1 dup key: { : "Web Threat Detection" }; nested exception is com.mongodb.MongoWriteException: E11000 duplicate key error collection: respond-server.aggregation_rule index: name_1 dup key: { : "Web Threat Detection" }2020-02-24 05:21:03,790 [ main] ERROR o.s.b.SpringApplication|Application run failedorg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'referenceDataLoader': Invocation of init method failed; nested exception is org.springframework.dao.DuplicateKeyException: E11000 duplicate key error collection: respond-server.aggregation_rule index: name_1 dup key: { : "Web Threat Detection" }; nested exception is com.mongodb.MongoWriteException: E11000 duplicate key error collection: respond-server.aggregation_rule index: name_1 dup key: { : "Web Threat Detection" } at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:139) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)

Resolution

If there is no custom aggregation rule, then you can simply remove all the records from aggregation_rule collection.
An aggregation_rule will be restored after restarting respond-server.

Follow the steps below to fix this issue.
(Note) ensure that you take backup your aggregation_rule using "mongodump" command before removing all the records.
  1. Connect to the NetWitness server by SSH as the root user and login to Mongo Database.
# mongo admin -u deploy_admin -p {Your_Password} 
  1. Switch to respond-server db and find aggregation_rule collection.
> use respond-server
switched to db respond-server
> db.aggregation_rule.find()
  1. Remove all the records from aggregation_rule collection and make sure it is empty.
> db.aggregation_rule.remove({})
> db.aggregation_rule.find()
> exit
  1. Restart Respond service.
# systemctl stop rsa-nw-respond-server.service
# systemctl start rsa-nw-respond-server.service

Once completed, you are able to see that the respond-server came up/online and it is showing up in the GUI as well.

If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.3.x
Platform: Cent OS
O/S Version: 7

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue