Netwitness NodeX host services install fails due to invalid certificates.

Issue

The NodeX host services install fails with the below error.

/var/log/netwitness/config-management/chef-solo.log:

 [2022-09-02T14:49:42+00:00] ERROR: nw_base_nwconsole[refresh-nw-concentrator-trust-peers] (nw-concentrator::trusts line 31) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[list-peercerts] (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/nwconsole.rb line 134) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
 
---- Begin output of NwConsole -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q ----
 
STDOUT: Could not create trusted session: server could not validate and trust our certificate
 
STDERR:
 
---- End output of NwConsole -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q ----
 
Ran NwConsole -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q returned 1; ignore_failure is set, continuing
 
[2022-09-02T14:49:42+00:00] INFO: Processing systemd_service_drop_in[nwconcentrator-opts-managed] action create (nw-concentrator::services line 12)
 
[2022-09-02T14:49:42+00:00] INFO: Processing service[nwconcentrator] action enable (nw-concentrator::services line 22)
 
[2022-09-02T14:49:42+00:00] INFO: Processing service[nwconcentrator] action start (nw-concentrator::services line 22)
 
[2022-09-02T14:49:42+00:00] INFO: Processing log[Unable to find Common Node Certificate] action write (nw-concentrator::collectd line 12)
 
[2022-09-02T14:50:44+00:00] ERROR: Running exception handlers
 
[2022-09-02T14:50:44+00:00] ERROR: Exception handlers complete
 
[2022-09-02T14:50:44+00:00] FATAL: Stacktrace dumped to /var/lib/netwitness/config-management/cache/chef-stacktrace.out
 
[2022-09-02T14:50:44+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
 
[2022-09-02T14:50:44+00:00] FATAL: NoMethodError: nw_base_nwconsole[set_pin_dir_and_size for nw-concentrator] (nw-concentrator::pin line 8) had an error: NoMethodError: execute[configure_pin_dir] (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/nwconsole.rb line 214) had an error: NoMethodError: undefined method `split' for nil:NilClass

Cause

This issue is due to invalid certificates in NodeX host.

Note the NodeX UUID from the below command.
cat /etc/salt/minion

From Admin Server, Please run the below command to test the validity NodeX certificates.

salt ' ' cmd.shell runas=root cmd='hostname; NwConsole -q -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list"'

Example:

  [root@AdminServer ~]# salt '846c32b7-9dbc-445c-9cc8-4d6d696f2fab' cmd.shell runas=root cmd='hostname; NwConsole -q -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list"'
  
846c32b7-9dbc-445c-9cc8-4d6d696f2fab:
  
 LogHYBRID
  
  Could not create trusted session: server could not validate and trust our certificate
  
 (F) 2022-Sep-02 15:34:14 [ChannelManager::messageHandler] Socket Error: Operation canceled
  
ERROR: Minions returned with non-zero exit code

Resolution

Please follow the below steps to copy trusted certificates from AdminServer to NodeX.

Take backup of existing certificates from NodeX host using the below commands.

mv /etc/pki/nw/node/node-cert.pem /root/
mv /etc/pki/nw/node/node-key.pem /root/

Please download the below files from AdminServer and upload them to NodeX host in the same directories.

/etc/pki/nw/node/node-cert.pem
/etc/pki/nw/node/node-key.pem

Change the ownership of the files to netwitness:nwpki using the below commands in NodeX host.

cd /etc/pki/nw/node/
chown netwitness:nwpki node-cert.pem
chown netwitness:nwpki node-key.pem

Verify the validity of NodeX certificates by running the below command in Admin Server.

  [root@AdminServer]# salt '846c32b7-9dbc-445c-9cc8-4d6d696f2fab' cmd.shell runas=root cmd='hostname; NwConsole -q -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list"'
  
846c32b7-9dbc-445c-9cc8-4d6d696f2fab:
  
 LogHYBRID
  
 "f24e95c5.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = 3c95eea0-2561-426e-a9e8-3c2ada58f9ec
  
 sha-1:CF:6A:A2:B8:D4:17:90:28:66:F4:4E:3C:79:A7:6A:28:1A:A3:CC:CC
  
 "67342faa.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = nwappliance
  
 sha-1:BB:E0:8B:77:87:B0:D8:E5:54:53:38:F0:AD:39:8A:53:8F:56:A1:10
  
 "eeb9f4d7.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = 3c95eea0-2561-426e-a9e8-3c2ada58f9ec
  
 sha-1:CF:6A:A2:B8:D4:17:90:28:66:F4:4E:3C:79:A7:6A:28:1A:A3:CC:CC
  
 "1386a7d5.0" CN = rsa-nw-respond-server
  
 sha-1:78:2A:2F:92:20:3F:42:8E:54:B3:D6:98:A6:32:2E:38:BD:37:9A:0C
  
 "4349e381.0" CN = rsa-nw-metrics-server
  
 sha-1:49:0A:37:CA:10:6E:37:88:44:7C:77:07:5F:04:93:1F:8B:41:42:05

Try installing the services on the Hosts page, and it will be successful now.

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.x
Platform: CentOS 7

Summary

This document outlines the procedure to install the services with correct certificates.

Approval Reviewer Queue

Technical approval queue