NetWitness source-server service will not start after upgrade to version 12.3

Issue

The source server was not coming up post upgrade to 12.3

Cause

The reason behind this issue is serviceContentState collection in DB was overpopulated with stale data and we were not able to delete them while upgrading.

Workaround

1. We took a backup of MongoDB

 # mongodump --host localhost -d source-server --port 27017 -u deploy_admin --authenticationDatabase admin -o source-server

2. We logged in to MongoDB from the SSH session:

 # mongo admin -u deploy_admin
 
> use source-server

3. We searched and counted the stale entry in the MongoDB contentLibrary collection

 > db.getCollection("contentLibrary").find({"custom":false, "path": null}).count()

4. We Stopped the source server from a different session

 # service rsa-nw-source-server stop

5. We deleted the stale entries from MongoDB

 > db.getCollection("contentLibrary").deleteMany({"custom":false, "path": null})

6. After that we started the source server and observed the log

 # service rsa-nw-source-server start && tailf /var/log/netwitness/source-server/source-server.log

After this, we see that the error logs are no longer occurring and source-server is syncing the contents, we have checked the content library to ensure all the contents are there.

Resolution

The solution to this is to remove all the stale entries that we failed to delete while upgrading manually.

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Netwitness Server
RSA Version/Condition: 12.x
Platform: CentOS O/S Version: 7

Summary

Upgrade was successful however the Source Server service is showing an error and did not start in the UI.

Approval Reviewer Queue

Technical approval queue