Core Services Appear Offline on UI after 12.1 Upgrade

Issue

Core Components go offline on UI after the upgrade to NetWitness version 12.1

Cause

The test connection on UI requires both the admin-cert.pem and sa-server certificates to be present in the core service's trust peer certificates directory (/etc/netwintess/ng/ /trustpeers/). Prior to version 12.1, these two certificates were practically the same, so having one of them in the directory was enough. Since 12.1, the two are no longer the same.

The upgrade updates both of the certificates on the Admin Node, they are then both transferred to the other hosts (you can find them in /etc/pki/nw/peer/), however only the sa-server certificate is added to the core service's trust-peers folder, which causes the service to appear offline on UI.

To check if this is the case, you can run the following command on both the Admin Server's Broker and any of the offline hosts's core service:

  NwConsole -k -c "tlogin server=localhost port=5600x username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q

Note: replace the 'x' with the correct port for the service (eg. 56003 for Broker, 56001 for Log Collector, etc.)

The expected output should look like the below:

On Admin Server:

  NwConsole -k -c "tlogin server=localhost port=56003 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q
  
"67342faa.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = nwappliance
  
sha-1:58:C0:54:6E:C2:CF:5A:16:4E:EB:5F:3E:62:B2:1E:A3:98:D6:BA:BA
  
"cf280d67.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = rsa-nw-sa-server
  
sha-1:C3:B1:E2:C4:7B:FE:1E:7D:5B:52:94:1C:C1:D0:12:9D:10:AA:1C:0E
  
"1386a7d5.0" CN = rsa-nw-respond-server
  
sha-1:C7:CF:6D:82:20:26:57:5C:86:12:80:F2:F9:2B:78:19:12:AB:12:84
  
"4349e381.0" CN = rsa-nw-metrics-server
  
sha-1:84:4F:63:26:02:5D:10:AB:56:6A:C0:8C:D4:DB:A5:E9:8E:88:E9:3F
  
"4349e381.1" CN = rsa-nw-metrics-server
  
sha-1:53:D2:9A:5F:34:88:08:45:96:86:55:C3:3D:05:7C:44:6B:45:AA:C7
  
"e50d1735.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, 
  CN = 26984ff7-4c91-4f07-9cbc-39b95923f5d8
sha-1:FE:8A:69:7E:78:59:B6:56:8E:A9:13:5E:C7:24:00:0E:0C:6F:FA:46

On any other host:

  NwConsole -k -c "tlogin server=localhost port=56001 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q
  
"d5c1895c.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = 6595bb5e-b08f-4b33-9ac6-051c24e478fa
  
sha-1:7C:D9:F3:92:AB:BC:26:8E:73:89:F2:03:18:13:EF:D8:78:62:8B:79
  
"67342faa.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = nwappliance
  
sha-1:EC:A5:6F:C8:09:73:1A:17:37:B4:FE:7C:42:A1:63:25:94:D0:AA:CC
  
"cf280d67.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = rsa-nw-sa-server
  
sha-1:C3:B1:E2:C4:7B:FE:1E:7D:5B:52:94:1C:C1:D0:12:9D:10:AA:1C:0E
  
"4349e381.0" CN = rsa-nw-metrics-server
  
sha-1:84:4F:63:26:02:5D:10:AB:56:6A:C0:8C:D4:DB:A5:E9:8E:88:E9:3F
  
"4349e381.1" CN = rsa-nw-metrics-server
  
sha-1:53:D2:9A:5F:34:88:08:45:96:86:55:C3:3D:05:7C:44:6B:45:AA:C7
  
"e50d1735.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, 
  CN = 26984ff7-4c91-4f07-9cbc-39b95923f5d8
sha-1:C9:25:9F:88:77:4F:3C:FB:91:5B:B6:2D:79:54:92:9E:8A:74:91:D0

Workaround

A) For All Core Services (except Windows Legacy Collector 'WLC'):

In order to fix the issue, the updated admin-cert.pem must be added to the core service's trust peers. The following command copies the new admin-cert.pem from the Admin Server to all the hosts at /etc/pki/nw/peer/, creates the corresponding symlink in /etc/netwitness/ng/ /trustpeers/, then restarts the core services on all hosts for the changes to take effect.

On the Admin Server, run the the following:

  salt-cp '*' /etc/pki/nw/peer/admin-cert.pem /etc/pki/nw/peer/ && salt '*' cmd.run 'ln -s /etc/pki/nw/peer/admin-cert.pem /root/templink ; find /etc/netwitness/ng/ -name "trustpeers" -exec cp -av /root/templink {}/"$(openssl x509 -hash -in /etc/pki/nw/peer/admin-cert.pem -noout).0" \; && rm -vf /root/templink' && salt '*' cmd.run 'systemctl restart nwappliance nwlogdecoder nwdecoder nwlogcollector nwarchiver nwworkbench nwconcentrator nwbroker'
 

After running the command:

  [root@CS-NWVLC-48 trustpeers]# NwConsole -k -c "tlogin server=localhost port=56001 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q
  
"d5c1895c.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = 6595bb5e-b08f-4b33-9ac6-051c24e478fa
  
sha-1:7C:D9:F3:92:AB:BC:26:8E:73:89:F2:03:18:13:EF:D8:78:62:8B:79
  
"67342faa.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = nwappliance
  
sha-1:EC:A5:6F:C8:09:73:1A:17:37:B4:FE:7C:42:A1:63:25:94:D0:AA:CC
  
"cf280d67.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = rsa-nw-sa-server
  
sha-1:C3:B1:E2:C4:7B:FE:1E:7D:5B:52:94:1C:C1:D0:12:9D:10:AA:1C:0E
  
"4349e381.0" CN = rsa-nw-metrics-server
  
sha-1:84:4F:63:26:02:5D:10:AB:56:6A:C0:8C:D4:DB:A5:E9:8E:88:E9:3F
  
"4349e381.1" CN = rsa-nw-metrics-server
  
sha-1:53:D2:9A:5F:34:88:08:45:96:86:55:C3:3D:05:7C:44:6B:45:AA:C7
  
"e50d1735.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, 
  CN = 26984ff7-4c91-4f07-9cbc-39b95923f5d8
sha-1:FE:8A:69:7E:78:59:B6:56:8E:A9:13:5E:C7:24:00:0E:0C:6F:FA:46

Once it matched the Admin Server's output, you can try testing the connection from UI and it should be successful.

B) For Windows Legacy Collector (WLC):

Since the WLC cannot be managed by salt, the process can be performed manually using the WLC's REST port.
1) Login to the WLC's REST page:
https:// :50101
2) Click on Sys
3) /Sys/trustpeer
4) Select Add, and paste the content of the Admin Server's /etc/pki/nw/peer/sa-server/ .pem file in the text box and click Upload
5) Perform the same step (4) for the /etc/pki/nw/peer/admin-cert.pem
6) Restart the nwlogcollector service
7) Test connection, it should be successful

You can copy the resulting certificate files to other WLCs to avoid performing the same steps on all of them.

Notes

The command will restart all the core services on all hosts. Make sure to stop the aggregation/capture on the hosts prior to running the command.

When running the NwConsole command, make sure to double check the two certificates having (CN= and CN=rsa-nw-sa-server) , they both should match the one on the Admin Server in order for the connection to be successful.

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Log Collector (Local/Remote), Log Decoder, Decoder, Broker, Concentrator, Archiver
RSA Version/Condition: 12.x, 11.7.x
Platform: CentOS O/S Version: 7

Summary

This is a quick workaround to the 12.1 upgrade issue that causes the core services to go offline on UI.

Approval Reviewer Queue

Technical approval queue