RSA NetWitness ESA Correlation or Contexthub Server services appear offline under Admin-SERVICES page.

Issue

After upgrading from an older version to 11.4.x or 11.5.x, ESA Correlation and/or Contexthub server service appear offline in the User Interface under Admin- SERVICES.
Test connection to the services fails consistently.

Running the following commands confirms that the service are running fine.
systemctl status rsa-nw-correlation-server
systemctl status rsa-nw-contexthub-server

The log files may show a connection error to the Admin Server's rabbitmq port(5671).
/var/log/netwitness/correlation-server/correlation-server.log
/var/log/netwitness/contexthub-server/contexthub-server.log

 2021-06-14 02:24:47,531 [Connection <Admin Server IP>:5671] ERROR c.r.c.i.ForgivingExceptionHandler|An unexpected connection driver error occured
 
java.net.SocketException: Socket is closed
 
 at com.rsa.sslj.x.ap.l(Unknown Source)
 
 at com.rsa.sslj.x.ap.b(Unknown Source) 
 
 at com.rsa.sslj.x.ap.b(Unknown Source) 
 
 at com.rsa.sslj.x.al.read(Unknown Source)

Running curl -v nw-node-zero:5671 proves no connectivity issue to the Admin Server's port 5671.

Cause

The issue may occur when the service's keystore or certificate is corrupted or not in sync.

Resolution

In order to resolve the issue, please perform the following to regenerate the required files.

From the ESA host -

If ESA Correlation service appears offline:
1. mkdir /root/backup_correlation
2. systemctl stop rsa-nw-correlation-server
3. mv /etc/systemd/system/rsa-nw-correlation-server.service.d/rsa-nw-correlation-server-opts-managed.conf /root/backup_correlation
3. cd /etc/netwitness/correlation-server/ && mv keystore.p12 lockbox.ss lockbox.ss.lock /root/backup_correlation
4. cd /etc/pki/nw/service/ && mv rsa-nw-correlation-server* /root/backup_correlation
5. mv /etc/pki/nw/service/bootstrap/correlation-server.completed /root/backup_correlation
6. cat /etc/salt/minion - note the node ID

If Contexthub Server service appears offline:
1. mkdir /root/backup_contexthub
2. systemctl stop rsa-nw-contexthub-server
3. mv /etc/systemd/system/rsa-nw-contexthub-server.service.d/rsa-nw-contexthub-server-opts-managed.conf /root/backup_contexthub
3. cd /etc/netwitness/contexthub-server/ && mv keystore.p12 lockbox.ss lockbox.ss.lock /root/backup_contexthub
4. cd /etc/pki/nw/service/ && mv rsa-nw-contexthub-server* /root/backup_contexthub
5. mv /etc/pki/nw/service/bootstrap/contexthub-server.completed /root/backup_contexthub
6. cat /etc/salt/minion - note the node ID

From the Admin Server -
orchestration-cli-client --refresh-host --host

Product Details

RSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: ESA Correlation and Contexthub server
RSA Version/Condition: 11.4.x, 11.5.x
Platform: CentOS
O/S Version: 7

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue