NetWitness Endpoint Server stopped accepting agent data and is in unhealthy state

Issue

Log Decoder service on Endpoint Log Hybrid shows 0 EPS indicating that no data is received from the Endpoint Agents configured for Windows collection.
'AGENT LAST SEEN' stats do not get updated under the Hosts page.
The logs are received, and the stats are updated for a very short period after Endpoint Server service is restarted but the issue continues soon after.

endpoint-server.log shows warnings like below.

Cause

The issue can occur when large documents containing FILELESS_SCRIPT exceeds the Mongo DB's limitation of 16 MB.
The attached test.js script can be used to determine the documents and their collection that exceeds 16 MB.

1. Download test.js to the Endpoint Server.
2. Modify test.js to replace netwitness with the actual deploy_admin password.
3. Execute the script.
   mongo test.js
4. Confirm that documents exceeding 16MB are returned as shown in the example below.
   ...
   Collection filecontexthistory documents: 4797384
   From Collection = filecontexthistory, document Exceeds Size = 605fa9348c4a4924bfe5b4fc
   From Collection = filecontexthistory, document Exceeds Size = 605fb376d638001695cbe1ea
   From Collection = filecontexthistory, document Exceeds Size = 6062424b336def5ba96163f2
   ...
   
    

Resolution

In order to resolve the issue, delete the documents that exceed 16 MB.

1. Back up the collection(s).
   mongoexport --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db endpoint-server --collection <collection_name> --out /root/<collection_name>.json
   e.g. mongoexport --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db endpoint-server --collection filecontexthistory --out /root/filecontexthistory.json
    
2. Delete every document noted in step 4 in Cause.
   mongo admin -u deploy_admin
   use endpoint-server
   db.<collection_name>.remove({"_id" : ObjectId("<The Returned IDs>")})
   
   e.g., db.filecontexthistory.remove({"_id" : ObjectId("605fa9348c4a4924bfe5b4fc")})
    
3. Exist the Mongo console and run 'mongo test.js' to confirm that no document is found to be exceeding the size.
    
4. Restart Endpoint Server service.
   systemctl restart rsa-nw-endpoint-server 

The issue is commonly noticed in environments where the Endpoint agents version is on 11.5.1 or older due to the bug which is fixed in 11.5.2 per ASOC-104219/SACE-14786. To avoid encountering the issue again, the agents need to upgrade to 11.5.2 or later.

Notes

If the issue persists, collect the following to investigate further.

1. Mongo outputs.
   mongo admin -u deploy_admin
   use endpoint-server
   db.file.count()
   db.file.find({firstFileName : /FILELESS_SCRIPT/}).count()
   db.filecontexthistory.count()
   db.filecontexthistory.find({firstFileName : /FILELESS_SCRIPT/}).count()
2. SOSReport after enabling the debugging for Endpoint Server service.
   From Endpoint Server service's Explore view, select logging, click on the field next to levels, and enter EndpointManagement:DEBUG,AgentManagement:DEBUG
   Collect SOSReport
3. Number of Endpoint agents deployed and its version.

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Endpoint Server
RSA Version/Condition: 11.5
Platform: CentOS
O/S Version: 7

Approval Reviewer Queue

Technical approval queue