Troubleshooting

Agent Last Seen Time column is not updated in the UI.

The issue could be due to any one of the following:

• Agent is inactive
• Agent data is not processed if the Endpoint.Health.Overall-Health statistic shows Unhealthy due to which all the agent data including agent last seen time is not updated.

See the resolution for these statistics in the Health and Wellness Issues section.

Agent is unable to communicate with the Endpoint Server.

This could be due to one of the following reasons:

• Agent is inactive.

• Endpoint Server settings is incorrect in the agent packager or policy configuration, or not available for communication.
• Endpoint Server or Nginx Server is not running .
• Firewall or IP table rules are blocking the connection between the host and Endpoint Server.

• Check if the Endpoint Server and Nginx Server are reachable.
• If the Endpoint Server settings are incorrect, uninstall the agent, download the agent packager, and reinstall the agent.
• Update firewall or IP table rules, if required.

Agent takes a long time to scan.

Sometimes, the NetWitness Endpoint scan takes a long time to complete. This is because of the CPU usage by other antivirus programs (such as Windows Defender, McAfee, Norton, and so on) that may be installed on the agent machines.

It is recommended to whitelist the (name provided in the packager, by default, the service name is NWEAgent.exe) file in the antivirus suite.

You want to change the responsiveness of the Agent.

Depending on your installation, you can adjust Beaconing intervals to change how responsive your agents are.

If resources are not a concern, you can lower the HTTPS Beacon Interval and UDP Beacon Intervals. If resources are a concern and responsiveness of the agent is not, you can increase these intervals.

• Run Services.msc and look for Windows Management Instrumentation (WMI) service.
• Go to properties and change the Startup type to Automatic.

Failed to load the client certificate.

Incorrect certificate password.

While generating the agent installer, the certificate password does not match with the one provided while downloading the agent packager from the UI.

Specify the correct certificate password.

Endpoint Server service or required resources are not available or not in a usable state. This could be due to one of the following reasons:

• Unable to forward Endpoint meta data to the Log Decoder.

• Endpoint Log Hybrid disk usage reaches the specified limit.

• Mongo DB is down or excessive read and write errors during processing.

• For Log Decoder issue, see Log Decoder Issues section.
• For disk usage and Mongo DB issues, see Disk Usage and Mongo Issues section.

The health check of the Data.Application.Connection-Health Application, Data Store Disk Usage or Data Persistence for Endpoint Server shows Unhealthy.

• Data.Application.Connection-Health Application or Data Persistence shows Unhealthy, if Mongo service is down or fails due to authentication.
• Data Store Disk Usage shows Unhealthy, if Endpoint Server Mongo storage size has exceeded the threshold. By default, the server automatically delete the old data when it reaches 80% of the disk space.

• For Data.Application.Connection-Health Application or Data Persistence issue, you must check the Endpoint server logs (/var/log/netwitness/endpoint-server/endpoint-server.log) and Mongo logs (/var/log/mongodb/mongod.log), and:
  • If the issue is due to authentication, you must reissue the certificate. For more information, see "Service Certificate Reissue" section in the System Maintenance Guide.
  • If the issue is due to Mongo service is down, you must restart the Mongo.
• For Data Store Disk Usage issue, you must increase the storage or configure data retention settings to clear the old data. For more information, see Configuring Data Retention Policy.

The health check of the Log Decoder Buffer and Meta Forward shows Unhealthy in the Health and Wellness.

The issue could be due to any of the following reasons:

• Log Decoder capture is not started.
• Concentrator aggregation is not started.
• Log Decoder connection issue.
• Log Decoder buffer usage is beyond the specified limit.

Make sure that:

• Capture is enabled on the Log Decoder.
• Aggregation is enabled on the Concentrator.
• Meta forwarding is configured properly.

Note: Make sure Capture Autostart is enabled in the Service Config view for Log Decoder and Aggregate Autostart is enabled in the Service Config view for Concentrator.

Policies can be invalid for a variety of reasons. Some examples:

• No sources found if the policy is enabled.
• Invalid or missing typespec file
• No destination is reachable for a file log policy event source type

Additionally, if capture is stopped on the destination Log Decoder, Endpoint Agents will send an error to the Endpoint Server saying that they failed to connect.

Also, if there is a lot of data to be processed for Agents collecting File data (when File Policy is enabled) , there is a possibility that Log Decoder buffer becomes full. If this happens, the Log Decoder cannot process any requests from the Agents communicating via EPS.

The system is dynamic in nature, which means its state can change: event sources can lose their connection, typespec files can be altered or deleted, and other changes can occur that can invalidate a previously valid policy.

To help identify the specific issue, check the log file on the Endpoint Server that reports the error:

/var/log/netwitness/endpoint-server/endpoint-server.audit.log

Relevant errors will be listed as FileLogError in the log file.

If you experience this issue, you can do the following:

1. Try to identify and target higher-value data, thus limiting the total amount of data being processed.
2. Enable throttling in the File policy to smooth out the peaks in usage.
3. If you really do need to process more data on a regular basis, consider server-side hardware upgrades.

If the system is not configured correctly, NetWitness might collect logs and not be able to parse them. Or, files might get sent, but for some reason, not make it to the Log Decoder (for example if communication is via UDP and there is a network connectivity issue).

In these and other cases, you can reprocess these "missing" log files.

For whatever reason, you may need to reprocess logs from the beginning of the file.

Reset bookmarks for an event source type using the procedure described here: Reset File Collection Bookmarks.

Some log collectors or event sources seem to be missing from the list of available items.

The Filter drop-down menus (types, log collectors, and log decoders) only show values that are in the event sources database, rather than all possible values. For example, if you have a log collector that has not yet collected any logs, then it is missing from the list.

Collect logs from a specific log collector and event source, and then they should appear as items in the appropriate menu.

Relay Server test connection failed.

1. Check if the hostname or IP and port of the Relay Server are correct.
2. Make sure that the hostname or IP of the Relay Server is resolvable from the Endpoint Server. Perform the following:
   1. In the Endpoint Log Hybrid console, verify if the Relay Server is reachable using the following command:
      nc -zvw3
      If the Relay Server is not reachable contact your Administrator.
   2. If the Relay Server is reachable, verify if the correct Relay Server installer is used by getting the Endpoint Server revision ID from the Relay Server host (/var/log/relay-install.log) and check the Endpoint Server RPM on Endpoint Log Hybrid using the following command:
      rpm -qa | grep
   3. Make sure if the Relay Server is installed and running.
      • Verify the Relay Server installation logs using the following command:
        /var/log/relay-install.log
      • Verify the status of Relay Server using the following command:
        systemctl status rsa-nw-relay-server

Test fails when installing relay server in cloud, using CentOS 7 configuration

1. Check if you have entered the suggested port numbers.
2. If you have entered any other port number than the suggested one
3. Make sure that the hostname or IP of the Relay Server is resolvable from the Endpoint Server. Perform the following:
   1. In the Endpoint Log Hybrid console, verify if the Relay Server is reachable using the following command:
      nc -zvw3
      If the Relay Server is not reachable contact your Administrator.
   2. If the Relay Server is reachable, verify if the correct Relay Server installer is used by getting the Endpoint Server revision ID from the Relay Server host (/var/log/relay-install.log) and check the Endpoint Server RPM on Endpoint Log Hybrid using the following command:
      rpm -qa | grep
   3. Make sure if the Relay Server is installed and running.
      • Verify the Relay Server installation logs using the following command:
        /var/log/relay-install.log
      • Verify the status of Relay Server using the following command:
        systemctl status rsa-nw-relay-server

You must retry the download after 5-10 minutes. If the download still fails even after all dependencies are downloaded in the Endpoint Server, contact the NetWitness Customer Support.

Note: You can check ‘Finished downloading all Relay Server dependencies’ message in the Endpoint Server logs at /var/log/netwitness/endpoint-server/endpoint-server.log, to see if the dependencies are downloaded. If the download fails due to yum related issues, then you must clean yum repo using the command yum clean all and restart the Endpoint Server.

After the removal of DNSMasq in 12.1 and later versions, test connection fails between the Endpoint server and relay server.

For the test connection to succeed, perform the following.

1. SSH to Endpoint server.

2. Edit the /etc/nginx/conf.d/relay.conf file and go to resolver nw-node-zero ipv6=off; line.

3. Replace nw-node-zero with the nameserver IP or hostname.

4. Run the following command to restart nginx service.

   systemctl restart nginx

5. Try Test Connection again.

Re-download the installer dependencies, perform the following:

1. Go to (Admin) > Endpoint Server service > select > View > Explore.
2. In the Endpoint server configuration, make sure endpoint.relay.installer.download-on-restart boolean is set to true (by default it is true).
3. Restart the Endpoint server using the following command:
   systemctl restart rsa-nw-endpoint-server
   Fresh dependencies will be downloaded to the local directory in the Endpoint Server. This may take few minutes.
4. Download the Relay Installer.
5. Run the Relay Server Installation Script.
   For more information, see (Optional) Installing and Configuring Relay Server.