NetWitness Upgrade Precheck : NwConsole Authentication Probe

Issue

While running either the standard NetWitness Pre-Upgrade Checklist or the Standalone 12.4 Pre-Upgrade Checklist, a warning is returned that references a missing KB or incorrect KB Number. The Title of the check resembles "NwConsole Authentication Probe Failed" and the incorrect KB it may reference is 000003159.

Cause

This can be caused by missing symbolic links from platform level certificates to individual service certificates.

Workaround

Take a backup of current certificate checksums and the certs themselves:

 sha256sum /etc/netwitness/ng/*/trustpeer*/* /etc/pki/*/*/* /etc/pki/nw/peer/*/* >> /root/cert_sha256sum.out
 
tar -cvf /root/certificate_backups.tar /etc/netwitness/ng/*/trustpeer*/* /etc/pki/*/*/* /etc/pki/nw/peer/*/* /root/cert_sha256sum.out

Verify the backup was created successfully:

 [root@NEW-NW11-NW-NODE-ZERO ~]# ls -lrth /root/certificate_backups.tar
 
-rw-r--r--. 1 root root 2.7M Oct 3 19:02 /root/certificate_backups.tar

Run the following commands in-order:

 ln -s /etc/pki/nw/peer/admin-cert.pem /root/templink && find /etc/netwitness/ng/ -name "trustpeers" -exec cp -av /root/templink {}/"$(openssl x509 -hash -in /etc/pki/nw/peer/admin-cert.pem -noout).0" \; && rm -vf /root/templink
 
counter=0; for file in /etc/pki/nw/peer/sa-server/*; do ln -s "$file" /root/templink && find /etc/netwitness/ng/ -name "trustpeers" -exec cp -av /root/templink {}/"$(openssl x509 -hash -in "$file" -noout).$counter" \; && rm -vf /root/templink; ((counter++)); done
 
counter=0; for file in /etc/pki/nw/peer/respond-server/*; do ln -s "$file" /root/templink && find /etc/netwitness/ng/ -name "trustpeers" -exec cp -av /root/templink {}/"$(openssl x509 -hash -in "$file" -noout).$counter" \; && rm -vf /root/templink; ((counter++)); done

Restart the appropriate core services for that host. In my example, I am running this on NW-NODE-ZERO (Admin Server), so the only core services I need to restart are the nwbroker and nwappliance services:

 [root@NEW-NW11-NW-NODE-ZERO ~]# systemctl restart nwbroker nwappliance

After the services are fully up and running, test Certificate Based NwConsole authentication against each
- The command is identical for all, but the port changes based off service. Here are the ports for the different services:
  - nwlogcollector = 56001
  - nwlogdecoder = 56002
  - nwbroker = 56003
  - nwdecoder = 56004
  - nwconcentrator = 56005
  - nwappliance = 56006
  - nwarchiver = 56008

 NwConsole -k -c "tlogin server=localhost port=56003 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send / ls"

Example of a successful output running against my Broker service:

 [root@NEW-NW11-NW-NODE-ZERO ~]# NwConsole -k -c "tlogin server=localhost port=56003 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send / ls"
 
 RSA NetWitness NextGen Console 12.3.1.0
 
 Copyright (c) 2001-2023, RSA Security LLC or its affiliates. All Rights Reserved.
 
 >tlogin server=localhost port=56003 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem
 
 Successfully logged in to localhost:56003 as session 823
 
 >send / ls
 
 169:0x0000000000100100 /broker/
 
 18:0x0000000000100600 /connections/
 
 157:0x0000000000100100 /index/
 
 19:0x0000000000100100 /logs/
 
 89:0x0000000000100100 /rest/
 
 125:0x0000000000100100 /sdk/
 
 99:0x0000000000100100 /services/
 
 100:0x0000000000100100 /storedproc/
 
 44:0x0000000000100100 /sys/
 
 2:0x0000000000100100 /users/

After resolving manually, re-run the NetWitness Preupgrade Checklist again to ensure it is no longer showing the NwConsole Authentication Probe failure.

Resolution

The resolution is to recreate these symbolic links, restart core services and test Certificate Based NwConsole Authentication manually before re-running the Pre-upgrade checklist to ensure the issue is no longer highlighted.

Notes

The standalone pre-upgrade check may have incorrectly referenced this KB as 000003159. I am including the old KB number so it can be searchable.

Internal Comments

Please do not approve until Technical Validation on SACE-22043 is complete. I am submitting this as a draft in the meantime so it can be searched internally.

Product Details

NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: All Core Components (excludes ESA/Correlation, Independent Endpoint Servers, UEBA)
NetWitness Version/Condition: 12.x
Platform: CentOS , AlmaLinux

Approval Reviewer Queue

Technical approval queue