Skip to content
  • There are no suggestions because the search field is empty.

Health-Check.sh Script to check resource consumption in a NetWitness environment

Issue

NetWitness already has the Health & Wellness service which provide a full overview for the health of all NetWitness Services and hosts, Yet I also created a script for a health check to perform a quick analysis on Disk usage, Memory utilization, Existence of Core files & If there were any failed services on any NetWitness host

Also, it lists all your hosts with regards to their Salt Minion IDs, hostnames, IPs and also provide a Salt Reachability check.


Resolution

How It Works:

The procedure actually consists of 2 scripts.

health-check.sh: This is a Script to run on the SA and it performs a simple Health-Check on your environment, it copies the Health-Check-host.sh to all hosts then turns it executable then run it at each host and provide the output and recommendation. It also lists all your hosts UUIDs "Salt Minion IDs", Hostnames & IPs and performs a Reachability Test as well.

health-check-host.sh: This script is copied to all NetWitness hosts when you run the health-check.sh on the SA. This script analyzes the host's disk usage, memory utilization, Existence of Core files & if there are any failed services on that host.

This script (health-check-host.sh) will not run manually, it will run once you run the health-check.sh script on the SA.

Instructions:

All Below steps are done on SSH session to the NetWitness Admin Server (SA).

  1. Using WinSCP, move the attached scripts to the Admin Server (SA) to be under /root
  2. You will only need to make the health-check.sh executable (not the health-check-host.sh)
    #chmod +x health-check.sh
  3. Run the health-check.sh
    #./health-check.sh

Sample Run: 

Sample
Sample


Notes

Note:

Minion did not return. [Not connected] OR No Response could point to one of the below reasons.

  1. If you are facing any network slowness and Salt Master (SA) is unable to reach to the Salt Minions (hosts) within a specific time limit during fetching their IPs, Hostnames, the output of the first part of the script can provide you (No Response), Do not Panic, This does not mean that SA is totally unable to reach the Host(s), but it was unable to reach it during a specific time limit thus the salt will temporarily provide you with "No Response" output. 

    If you run the script again during no network slowness, it should provide the output as expected.

  2. If the host is having 0 free memory left and utilized all its swap memory, its salt minion may not reply to salt master's request of IP, hostname & Reachability test; giving Minion did not return. [Not connected]. If you run the script again, it will show a result normally, otherwise, Thanks to checking the memory utilization of this host if you already isolated it is not related to above 1st point  (network slowness) or a retired host/powered off-host (3rd point)
  3. Minion did not return. [Not connected] could also point to a retired host (that was removed from the environment yet its salt minion UUID was not deleted from the salt master) or could point to a host that is currently powered off.

Please feel free to provide feedback, bug reports, etc.


Product Details

RSA Product Set: Security Analytics, NetWitness Platform
RSA Product/Service Type: Admin Server (SA)

Approval Reviewer Queue

Technical approval queue