NetWitness upgrade preparing failure with '401 Unauthorized'

Issue

Upgrade failure occurs during preparing step. Below is the error messages in orchestration-client.log

- /var/log/netwitness/orchestration-client/orchestration-client.log

 2021-12-20 01:17:23,473 [ main] INFO c.r.n.i.o.c.LaunchHelper|Task [prepare host '10.x.xx.xxx' for upgrade '11.5.x.x'] running (polling 580 more times)...
 
2021-12-20 01:17:29,500 [ main] INFO c.r.n.i.o.c.LaunchHelper|Task [prepare host '10.x.xx.xxx' for upgrade '11.5.x.x'] running (polling 579 more times)...
 
2021-12-20 01:17:35,521 [ main] ERROR c.r.n.i.o.c.OrchestrationClient|Task [prepare host '10.x.xx.xxx' for upgrade '11.5.x.x'] stopped with errors!
 
2021-12-20 01:17:35,522 [ main] ERROR c.r.n.i.o.c.OrchestrationApplication|Requested operation failed, aborting...

'Authentication' related errors occur in orchestration-server.log and /var/log/salt/master.

- /var/log/netwitness/orchestration-server/orchestration-server.log

 2021-12-20 01:17:32,594 [ Asynchronous Task Monitor] WARN TaskManagement|Failed to check isJobReady
 
org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized
 
 at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:81)

- /var/log/salt/master

 2021-12-20 05:26:39,307 [salt.auth :299 ][WARNING ][127139] Authentication failure of type "eauth" occurred.

Cause

Authentication failure of salt services causes this problem.

Cause 1: nwsaltuser was locked or password was changed.

Cause 2: pam configuration changes cause this issue.

Following errors may occur in messages.

 Dec 20 05:26:28 NWAPPLIANCE31005 python[127142]: PAM unable to dlopen(/usr/lib64/security/pam_fprintd.so): /usr/lib64/security/pam_fprintd.so: cannot open shared object file: No such file or directory
 
Dec 20 05:26:28 NWAPPLIANCE31005 python[127142]: PAM adding faulty module: /usr/lib64/security/pam_fprintd.so

Workaround

Workaround for Cause 1: nwsaltuser was locked or password was changed.

1. Check the nwsaltuser.

 # chage -l nwsaltuser

2. Reset the password of nwsaltuser and unlock.

 # security-cli-client --get-config-prop --prop-hierarchy nw.orchestration-server --prop-name rsa.orchestration.engine.salt.client.password -q | passwd --stdin nwsaltuser
 
# faillock --user nwsaltuser --reset
 
# passwd -u nwsaltuser
 
# chage -M -1 nwsaltuser

3. Restart salt services.

 # systemctl restart salt-master salt-api

Workaround for Cause 2: pam configuration changes cause this issue.

1. Check and confirm whether the user or third-party application made any changes in /etc/pam.d/system-auth file or other pam configurations.

2. Check /etc/pam.d/system-auth file has a symbolic link to /etc/pam.d/system-auth-stig file.

 # ll /etc/pam.d/system-auth
 
 correct output: lrwxrwxrwx. 1 root root 27 Oct 5 2020 /etc/pam.d/system-auth -> /etc/pam.d/system-auth-stig
 
 Incorrect(no symbolic link): lrwxrwxrwx. 1 root root 27 Oct 5 2020 /etc/pam.d/system-auth

3. (if system-auth have symbolic link to system-auth-stig, skip this step) Backup the system-auth file then make a symbolic link.

 # mv /etc/pam.d/system-auth /tmp
 
# ln -s /etc/pam.d/system-auth-stig /etc/pam.d/system-auth

4. Compare the system-auth file with the original system-auth file of OOTB of NetWitness, and revert to the original one if there're any differences.

- Example of system-auth of NetWitness version 11.5.x

 # cat /etc/pam.d/system-auth
 
#%PAM-1.0
 
# This file is auto-generated.
 
# User changes will be destroyed the next time authconfig is run.
 
auth required pam_env.so
 
auth required pam_faildelay.so delay=2000000
 
auth required pam_faillock.so preauth silent even_deny_root deny=3 unlock_time=never root_unlock_time=600 fail_interval=900
 
auth sufficient pam_unix.so try_first_pass
 
auth [default=die] pam_faillock.so authfail even_deny_root deny=3 unlock_time=never root_unlock_time=600 fail_interval=900
 
auth requisite pam_succeed_if.so uid >= 1000 quiet_success
 
auth required pam_deny.so
 
account required pam_faillock.so
 
account required pam_unix.so
 
account sufficient pam_localuser.so
 
account sufficient pam_succeed_if.so uid < 1000 quiet
 
account required pam_permit.so
 
password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
 
password sufficient pam_unix.so sha512 shadow try_first_pass use_authtok remember=5
 
password required pam_deny.so
 
session optional pam_keyinit.so revoke
 
session required pam_limits.so
 
-session optional pam_systemd.so
 
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
 
session required pam_unix.so

5. Restart salt services or reboot the Admin node(node-zero) appliance.

 # systemctl restart salt-master salt-api
 
  or
 
# reboot

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.4, 11.5, 11.6, 11.7
Platform: Centos 7

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue