Skip to content
  • There are no suggestions because the search field is empty.

RSA NetWitness Admin Server fails to enable a host with salt minion/master errors

Issue

Running upgrade-cli-client command failed within few seconds without any specific error while upgrading a host from 11.6.0.0 to 11.6.0.1.
No event is logged in /var/log/netwitness/config-management/chef-solo.log on the target host.

/var/log/messages showed the salt-minion service failing to connect to the Salt Master.
Aug 13 01:17:02 NWVLC salt-minon: [ERROR ] Minion unable to successfully connect to a Salt Master.
curl -v nw-node-zero:4505 and curl -v nw-node-zero:4506 confirmed no connectivity issue to node zero.

Re-provisioning the host by following KB36443 failed as the enabling process for the discovered host appeared to be stuck.

/var/log/salt/master.log showed messages like below.
2021-08-13 02:18:17,010 [salt.transport.mixins.auth:138 ][ERROR ][121471] AES key not found
...
2021-08-13 02:52:16,859 [salt.master :1365][WARNING ][121453] Salt minion claiming to be 39a63bb1-1523-45bb-8bc1-a38d6a884b12 attempted to communicate with master, but key could not be read and verification was denied.
...
2021-08-13 03:19:10,590 [salt.transport.mixins.auth:388 ][INFO ][121471] Authentication failed from host 39a63bb1-1523-45bb-8bc1-a38d6a884b12, the key is in pending and needs to be accepted with salt-key -a 39a63bb1-1523-45bb-8bc1-a38d6a884b12


Cause

The issue can occur when /etc/hosts of the target host contain an incorrect/not current hostname for the localhost.

For example, having the old host name, SAVLC, in /etc/hosts can cause the Salt Master service to deny the authentication request from the Salt Minion service.
127.0.0.1 SAVLC localhost localhost.localdomain localhost4 localhost4.localdomain4 39a63bb1-1523-45bb-8bc1-a38d6a884b12
::1 SAVLC localhost localhost.localdomain localhost6 localhost6.localdomain6
...
10.10.7.15 39a63bb1-1523-45bb-8bc1-a38d6a884b12 39a63bb1-1523-45bb-8bc1-a38d6a884b12.netwitness NWVLC


Resolution

In order to resolve the issue, please ensure /etc/hosts, /etc/hosts.netwitness and /etc/hosts.user contain the current hostname.
After the files are being updated, run nwsetup-tui and discover/enable the host.

If the issue continues, try the steps below and discover again or follow KB36443 to completely remove the host and start fresh.
  • Move the /etc/salt/pki/minion/minion_master.pub file to the /tmp directory.
    mv /etc/salt/pki/minion/minion_master.pub /tmp
  • Restart salt-minion with the command below.
    systemctl restart salt-minion
  • Run nwsetup-tui.
    nwsetup-tui

     

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.6.0.1
Platform: CentOS
O/S Version: 7

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue