Endpoint Server shows offline when seen in the Investigate Hosts tab in RSA NetWitness Platform

Issue

The Endpoint Server shows the status as yellow in health and wellness for the endpoint service only, and in the UI when navigating to the Investigate>Hosts page it shows the Endpoint Server as offline.

Cause

The cause of this issue is the config server. The address of the Endpoint server that is saved in the mongodb gets cleared (possibly during a reboot of the endpoint). During orchestration, the Admin server's mongodb gets updated with the Event Stream Analysis's IP address instead of the Endpoint servers. This causes the system to try and connection to the ESA mongodb, and it is rejected. Thus the UI shows the Endpoint server offline in some investigate pages.

Resolution

The resolution is to perform the following steps:

1. On the Endpoint Server run the following command and replace with the Admin server's Node ID:

 security-cli-client --get-config-prop --prop-hierarchy nw --prop-name rsa.data.application.servers[0] -b 
 <AdminServer-NODE-ID>

The output should be the Event Stream Analysis's (ESA) NODE ID:

 Retrieving value from Config-Server for property: nw:rsa.data.application.servers[0] 
 <ESA-NODE-ID>

2. On the Admin Server run the following command:

 security-cli-client --get-config-prop --prop-hierarchy nw --prop-name rsa.data.application.servers[0]

The output should be the ESA's NODE ID

 Retrieving value from Config-Server for property: nw:rsa.data.application.servers[0] 
 <ESA-NODE-ID>

3. On the Admin server run the following command to get the prop-identity needed for the next command:

 cat /etc/netwitness/endpoint-server/service-id

4. On the Admin server run the following command with the prop-identity retrieved from the previous command:

 security-cli-client --get-config-prop --prop-name rsa.data.application.servers[0] --prop-identity 
 <EP's Service ID> -b 
 <AdminServer-NODE-ID>

The output of the Endpoint's Node ID will be as follows. If this output does show the correct Endpoint Node ID, stop and contact NetWitness Support as the the fix is beyond the scope of this article. However, if the NODE ID provided by this output is not the correct one for the Endpoint server, proceed to the next step.:

 Retrieving value from Config-Server for property: 
 <prop-identity>:rsa.data.application.servers[0]
 <EP-NODE-ID>
 

5. On the Admin server run the following command:

 security-cli-client --set-config-prop --prop-name rsa.data.application.servers[0] --prop-value 
 <EP-Node-ID> --prop-identity 
 <EP-Service ID> -b 
 <AdminServer-Node-ID>

6. On the Endpoint server run the following to restart the service.

 systemctl restart rsa-nw-endpoint-server.service

7. In the UI go to the Hosts page and reload the page a few times.

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Endpoint
RSA Version/Condition: 11.3.x, 11.4, 11.5, 11.6, 11.7, 12.0, 12.1
Platform: Linux

Approval Reviewer Queue

Technical approval queue