Skip to content
  • There are no suggestions because the search field is empty.

ESA-Mongo-stats script for displaying info about ESA Correlation Server and Respond-Server Performance in NetWitness

Issue

Whenever the below is reported:
  1. Delay in receiving ESA Alerts.
  2. Respond-Server stopped generating new alerts.
  3. Time difference between alert time and event time (either the alert time lags the event time or even sometimes, alert time is from the future).

The below could be reasons.
  1. Respond-Server Mongo DB is piled up not having enough space to generate new alerts.
    • Mainly that is because Respond Alert Retention is not configured thus there are old alerts that are stored in the Respond-Server not giving a chance for new ones to be generated or gets generated after some time with a lag due to slowness in Respond-Server processing.
  2. Any Delay or Time Difference may point out some main reasons.
    • Sessions behind on the Data sources themselves (to be checked on the data sources directly ex: Concentrator)
    • Sessions behind on the ESA itself due to lack of Memory/CPU so Esper Engine does not have enough chance to process real-time events against ESA Rule (Lack of memory or CPU could be due to a busy/complex ESA Rule which reserves most of ESA resources) 
    • Time Difference could also be due to a difference in the pointer (represented in epoch time) that ESA uses to fetch data from the data sources.

Resolution

Script Output:

ESA-Mongo-stats script is created which provides below information to ease the troubleshooting.

It runs on the ESA host and displays the below.
  1. ESA Mongo Databases >> Where you can check the Respond-Server size
  2. Total Number of Incidents stored in the Respond-Server.
  3. Total Number of Alerts that are stored in the Respond-Server  >>> This Can help along with above point 1 and below point 5 to check if Respond Alert Retention is needed or not.
  4. ESA Rules/Modules sorted in descending order with regards to the total number of alerts each is firing >>> can point out to busy ESA rules.
  5. Display the oldest Alert stored in Mongo DB.
  6. Display the most recent Alert stored in Mongo DB.
  7. Display the oldest Incident stored in Mongo DB.
  8. Display the most recent Incident stored in Mongo DB.
    Then the Script connects to the Admin server to fetch.
  9. Position Tracking per each data source in the ESA Deployment(s) to check if the pointer in real-time with regards to ESA time or not (DOC-53338 will help you with that).

How to deploy:
  1. With WinSCP, move the attached ESA-Mongo-stats-v1-5.sh script be under /root in your ESA host.
  2. Give Executable permission to the script.
    #chmod +x ESA-Mongo-stats-v1-5.sh
  3. Run the script.
    #./ESA-Mongo-stats-v1-5.sh

If the output is large, you can direct it to a file for your further analysis.

Notes

The script will need the root password when it SSH to SA (nw-node-zero).


Internal Comments

The script is modified to V 1.5 (fixed code bugs and enhanced displayed output).


Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Event Stream Analysis, Correlation Server, Admin server
RSA Version/Condition: 11.x

Summary

Bash Script to help fetch some useful information about the ESA Correlation Server and Respond-Server Performance.


Approval Reviewer Queue

Technical approval queue