RSA NetWitness aggregation and collectd fail when not all hosts have been upgraded
Issue
Concentrator sessions are falling behind, and aggregation fails to start after upgrading only the RSA NetWitness admin server.Aggregation does not start and collectd keeps failing to send its stats.
/var/log/messages reports:
collectd[4335]: NgNativeReader_NwConcentrator-FastUpdate: nwsdk failure: NwOpen returned 0; code 125; error: Connect failed with error "Operation canceled"; thread 4356
On the admin server:
- SSH as root
- Run:
orchestration-cli-client --refresh-host -o <salt_node_id>
Where the
would be found after id: on the concentrator when running:
When this fails, you will see in /var/log/netwitness/config-management/chef-solo.log something similar to this:
cat /etc/salt/minion
When this fails, you will see in /var/log/netwitness/config-management/chef-solo.log something similar to this:
[2020-05-15T18:23:07+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-05-15T18:23:07+00:00] ERROR: yum_package[rsa-nw-security-cli] (nw-pki::packages line 11) had an error: Chef::Exceptions::Package: Version ["11.3.1.0-1905061831.1.b951110.el7"] of ["rsa-nw-security-cli"] not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)
[2020-05-15T18:23:07+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2020-05-15T18:23:07+00:00] ERROR: yum_package[rsa-nw-security-cli] (nw-pki::packages line 11) had an error: Chef::Exceptions::Package: Version ["11.3.1.0-1905061831.1.b951110.el7"] of ["rsa-nw-security-cli"] not found. Did you specify both version and release? (version-release, e.g. 1.84-10.fc6)
[2020-05-15T18:23:07+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Cause
Unable to find the proper rsa-nw-security-cli package that the admin server is using to properly connect to the concentrator.
Resolution
Upgrade the concentrator and the rest of the hosts to the upgraded version being used by the admin server.
Product Details
RSA Product Set: RSA NetWitness PlatformRSA Product/Service Type: Security Analytics Server, concentrator
RSA Version/Condition: 11.3.1.1
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue