Troubleshoot ESA

Troubleshoot ESATroubleshoot ESA

This section describes common issues that may occur while using ESA, and it suggests common solutions to these problems.

Troubleshoot ESA Correlation ServicesTroubleshoot ESA Correlation Services

Problem:
On the NetWitness Platform Dashboard, the ESA service appears in red to indicate it is offline.

In the (Configure) > ESA Rules view, the following message appears: "The Service is either offline or not reachable."
Possible Causes: Several
Solutions:
When an ESA Correlation service is offline, there are many possible causes. However, a common issue is that you have created a rule that uses excessive memory and causes the ESA service to fail. To troubleshoot this problem, see Steps to Troubleshoot Memory Issues with an ESA Service Offline.

Other common causes might be that your firewall is blocking the connection between the ESA and NetWitness Platform, or the ESA Correlation service machine may be down.

To bring up ESA Services:

Go to (Admin) > Services, select your ESA service, and then select > Start.

If your ESA service is stopping and restarting in a loop, you may need to call Customer Support to get the services to start.

Problem:
On the NetWitness Platform Dashboard, the ESA service appears in red to indicate it is offline.

In the (Configure) > ESA Rules view, the following message appears: "The Service is either offline or not reachable.
Possible Causes: Configuration issues
Solutions:
If your system has been recently upgraded, you may have made a configuration error.

Go to (Admin) > Services, select your ESA service and then select > Edit. In the Edit Service dialog, click Test Connection. If the connection fails, you likely have a configuration error. Attempt to fix your configuration error and try again.

Problem: Suddenly, an ESA Correlation service is completely unresponsive and appears to have crashed. The ESA Correlation Log file shows a Too Many Open Files error.
Possible Causes: Connectivity issues
Solutions:
The most likely cause for this error is a connectivity issue between the ESA Correlation service and a data source used in an ESA rule deployment, such as a Concentrator or Decoder.
1. Restart the ESA Correlation service.
2. Go to (Admin) > Services, select your ESA service and then select > Restart.
3. Check the connectivity from ESA to the data source. Look for connection errors in the ESA Correlation logs. You can use SSH to get in the system and go to: /var/log/netwitness/correlation-server/correlation-server.log.
For example:

Error: com.rsa.netwitness.streams.
RecordStreamException: admin@ :56005::
com.rsa.netwitness.streams.RecordStreamException:
connect::com.rsa.asoc.transport.nw.session.
NextgenException: Failed to connect to
:56005

If there are network connectivity issues, fix the issues and then restart the ESA Correlation service again to see if it fixes the problem.

If you have a data source with intermittent connectivity, you should remove it from the ESA rule deployment.

Troubleshoot RSA Live Rules for ESATroubleshoot RSA Live Rules for ESA

Problem: I imported a group of rules from RSA Live, and now my ESA service is crashing. Why?
Possible Causes: You may not have configured the parameters for the RSA Live rule to tune it for your environment.
Solutions:
Each rule in RSA Live has a description that includes the parameters you must configure and prerequisites for your environment. Review this description to see if the rule is appropriate for your environment.

To ensure that you deploy rules safely in your environment, configure new rules as trial rules to test them in your environment. Trial rules add a safeguard for testing new rules. For details on this, see Deploy Rules as Trial Rules.

Problem:
I imported a group of rules from RSA Live, and while the rules deployed without errors, they were later disabled.
Possible Causes:
Not all RSA Live rules are meant for every environment. You may not have the correct meta in your ESA for the rule to run.
Solutions:
You can verify that a rule was disabled by going to (Configure) > ESA Rules > Services > Deployed Rule Stats. If the rule is disabled, the green icon does not display next to the rule.

If a rule deployed correctly but was disabled, check the logs for exceptions related to the rule. Specifically, check to see if the rules were disabled due to missing meta. To do this, go to the ESA Correlation logs. You can use SSH to get in the system and go to: /var/log/netwitness/correlation-server/correlation-server.log.

Then, search for a message similar to the following:

"Property named ‘ ' is not valid in any stream"

For example, you might see:

Failed to validate filter expression '(medium=1 and streams=2 or medium=3...(238 chars)': Property named 'tcp_flags_seen' is not valid in any stream

If a similar message displays, you may need to add a custom meta key to the Log Decoder or Concentrator. To do this, follow these instructions: "Create Custom Meta Keys Using Custom Feed" in the Decoder and Log Decoder Configuration Guide.

Troubleshoot ESA RulesTroubleshoot ESA Rules

Problem:
I have an ESA rule that is not getting deployed and is not creating alerts.
Possible Causes:
A meta key that the rule uses is a string array type, but it shows as a string type on ESA.
Solutions:
Check to see if any string array meta keys that the rule uses are configured as string array types on ESA. Go to (Configure) > ESA Rules > Settings tab (Meta Key References).
- If it shows string[], it is configured as a string array type on ESA. This is fine.
- If it shows string without the brackets, it is configured as a string type and you need to fix it on ESA.
In the ESA Correlation service Explore view, go to correlation/stream. Add string array meta keys to the multi-valued list to allow them to be used as an array in ESA rules. Go back to the Meta Key References and click the refresh icon (). Verify that the meta keys with a string array type show a value of .string[]. For additional details, see "Configure Meta Keys as Arrays in ESA Correlation Rules" in the ESA Configuration Guide.

Problem:
I set up notifications for a rule, but we are not receiving them. The correlation-server.log file does not show any errors. Why?
Possible Causes:
Correlation-server successfully sent the notification messages to integration-server, but when integration-server tried to send the notifications to their destination, it failed.
Solutions:
When troubleshooting notifications, check both the ESA Correlation service log files (/var/log/netwitness/correlation-server/correlation-server.log) AND the Integration-Server log files on the NetWitness Server (/var/log/netwitness/integration-server/integration-server.log). For an example, see Integration-Server SMTP Notification Error Example.

Note: For any notification-related troubleshooting, check the integration-server log file in addition to the log file of the service creating the notification.

Problem:
I created a rule with an enrichment, added an SMTP notification, and deployed my rule. We are not receiving SMTP notifications. Why?
Possible Causes:
You do not have a template that met the criteria to parse the events.
Solutions:
Check the ESA Correlation service log files to see if the SMTP notification failed: /var/log/netwitness/correlation-server/correlation-server.log. For more details on the notification error, check the Integration-Server log file on the NetWitness Server (also known as Node 0, Admin server, or NWServer): /var/log/netwitness/integration-server/integration-server.log.

If you use an ESA rule that has an enrichment, such as a Context Hub list, you must create a custom template. You can duplicate a default template and adjust it for your enrichment. See SMTP Notification Error Example below for a notification error example.

For information on creating a custom ESA template, see "Define a Template for ESA Alert Notifications" in the System Configuration Guide.

Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.

Problem: I created a custom rule (via the Rule Builder or Advanced EPL), and my rule is not firing. Why?
Possible Causes: You may have connectivity issues.
Solutions:
Check the Offered Rate statistic on the (Configure) > ESA Rules > Services tab. Select the ESA service and then look at the statistics on the tab for the Deployment.

If the Offered Rate is zero, then the ESA service is not receiving data from Concentrators. Check the ESA Correlation log files for connectivity issues: /var/log/netwitness/correlation-server/correlation-server.log.

If the offered rate is not zero, the meta key name and type used in the rule likely doesn't match the meta key present in events. Check to see if the meta key name and type used in the rule is valid by searching for the meta key name in (Configure) > ESA Rules > Settings tab (Meta Key References).

Problem: I created a custom rule (via the Rule Builder or Advanced EPL), and my rule is not firing. Why?
Possible Causes: There may be a problem with the rule.
Solutions:
If a specific rule is not firing, go to (Configure) > ESA Rules > Services to see if the rule was disabled. In the Deployed Rule Stats section, a rule that is disabled displays a clear enabled button (instead of the green enabled button).

You can also check Events Matched field. Go to (Configure) > ESA Rules > Services. From there, you can see the number of events that were matched in the Events Matched column.

If no events matched, check the logic of your rule for errors. For example, check the syntax for uppercase and lowercase errors, and check the time window. If the rule still doesn't fire, consider simplifying the logic of the rule to see if it fires when there is less complexity.

Problem: After a recent upgrade, I am not seeing alerts and I am seeing disabled rules.
Possible Causes: There may be a problem with the ESA rule deployment.
Solutions:
Deploy the ESA rule deployments again. Create a Deployment steps in the Live Services Management Guide provides more information on deploying rules using the ESA Correlation service.

If this does not resolve the issue, check the ESA Correlation log files for more information: /var/log/netwitness/correlation-server/correlation-server.log.

Problem:
After an update or upgrade to 11.3.0.2 or later, if I try to make an adjustment to some rules, I get an error when trying to save them.
Possible Causes:
The Ignore Case option may be selected for a meta key that does not contain alphabetic values, such as IP address.
Solutions:
In NetWitness Platform 11.3.0.2 and later, the Ignore Case option has been removed from the ESA Rule Builder - Build a Statement dialog for meta keys that do not contain text values. Adding Ignore Case on meta keys which do not contain alphabetic values causes additional processing to occur for no added benefit.

In the ESA Rule Builder - Build a Statement dialog, check to see if you have any meta keys that do not contain alphabetic characters, for example, ip_src and ip_dst. If you do, clear the Ignore Case checkbox for those meta keys and try to save the rule again.

Problem: After an upgrade to 11.3.0.2 or later, I see a warning message in the ESA Correlation service log file showing a difference between the multi-valued and default-multi-valued parameter meta key values. Why?
Possible Causes: You do not have the required meta keys on ESA Correlation that the Endpoint, UEBA, and Live content rules need to work.
Solutions: If you want to use the latest Endpoint, UEBA, and Live content rules, add the necessary meta keys to the multi-valued and single-valued parameter fields. For detailed information and instructions, see "Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys" in the ESA Configuration Guide.
For example warning messages, see Example ESA Correlation Server Warning Message for Missing Meta Keys.

Problem: Meta keys marked as sensitive for data privacy are still included in notifications and alerts for some rules.
Possible Causes: In ESA rules that do not select every piece of meta from the session (that is, using select *), you may see that data privacy (if enabled) and the Pivot to Investigate > Navigate link accessed from a context tooltip in the Respond Incident Details view does not work.
Solutions:
The following steps apply to all released versions of NetWitness 11.3 and later. In 11.4 and later, you do not need to follow these steps for data privacy. However, you need to follow these steps if you want to enable the Pivot to Investigate > Navigate link accessed from a context tooltip in the Respond Incident Details view.
1. Add the ESA generated event_source_id meta key to the index-concentrator-custom.xml file.
2. Add the event_source_id meta key to the SELECT statement within any ESA rule that does not select every piece of metadata from the session.
3. After the Concentrator changes take effect, redeploy the ESA rule deployment that contains the ESA rule.
To do this, see Update any ESA Rule that Selects Only Certain Meta Keys from the Session to Include event_source_id.

For NetWitness 11.4 and later, to resolve the data privacy issue, see How to Remove Sensitive Meta Keys Globally from All Alerts.

Problem:
The Pivot to Investigate > Navigate link does not work in a context tooltip accessed from Respond.
Possible Causes:
In ESA rules that do not select every piece of meta from the session (that is, using select *), you may see that data privacy (if enabled) and the Pivot to Investigate > Navigate link accessed from a context tooltip in the Respond Incident Details view does not work.
Solutions:
For NetWitness 11.3 and later (including 11.4), see Update any ESA Rule that Selects Only Certain Meta Keys from the Session to Include event_source_id.

Problem: I added a data source filter to the data sources in my ESA rule deployment. It was working fine and then all of a sudden, I stopped receiving alerts.
Possible Causes: If there are any adjustments to an application rule on the Decoders that are mapped to the data sources used in your data source filter, the filter does not work for that rule since the application rule used in the filter no longer exists.
Solutions:
The optional data source filter is available in NetWitness 11.5 and later.

If an application rule linked to a data source filter is modified on a Decoder, the filter must be removed, added again, and redeployed. The changes take effect on ESA after the deployment is redeployed.

Problem:
I added a data source filter to the data sources in my ESA rule deployment and I see an "Invalid header size" error while communicating with Core services in the ESA Correlation log file.
Possible Causes:
You are filtering out a large portion of the traffic and less sessions match the aggregation criteria.
Solutions:
In the Explore view node list for an ESA Correlation service, select correlation > stream. Decrease the max-sessions parameter from 10,000 to a lower session count and restart the ESA Correlation service. For step-by-step instructions, see .

Problem: I created a rule using a Context Hub list and it was working properly, but suddenly, the rule stopped firing and ESA is not processing any rules.
Possible Causes: If a Context Hub list that is used by ESA Correlation is renamed or deleted, ESA will not be able to access the list and may halt processing for all rules.
Solutions:
Do not rename or delete a Context Hub list that is used in a deployed ESA rule. ESA Correlation will also not be able to access a Context Hub list if you delete it and then add it again with the same name while the rule is deployed.

If you rename a Context Hub list or recreate the Context Hub list with the same name, update the ESA rules that use that Context Hub list, and then redeploy the ESA rule deployments that contain those rules.

SMTP Notification Error ExampleSMTP Notification Error Example

The following SMTP notification error example is an excerpt from a correlation-server.log file, which shows an error message for sending notifications with unsupported templates. In this example, there is a rule that is configured with the GeoIP enrichment, which has a hash table as one of its fields (the GeoIPLookup meta). Because the default SMTP template is only designed to deal with metas that are either singular values or arrays that contain only singular values, such as "ip.src":"1.1.1.1" and "action":["fw:inbound-network-traffic"], sending the email notification fails due to the array containing a hash table.

FTL stack trace ("~" means nesting-related):

- Failed at: ${value!""} [in template "smtp.ftl" in macro "value_of" at line 1, column 152]

- Reached through: @value_of metadata[key] [in template "smtp.ftl" at line 85, column 141]

----

...

For "${...}" content: Expected a string or something automatically convertible to string (number, date or boolean), or "template output" , but this has evaluated to an extended_hash (LinkedHashMap wrapped into f.t.DefaultMapAdapter):

==> value!"" [in template "smtp.ftl" at line 1, column 154]

Integration-Server SMTP Notification Error ExampleIntegra , , which shows a failure when the Integration-server attempts to send an email notification to the email notification server. In this case, you should check the email notification server configuration in the Global Notifications settings ( (Admin) > System > Global Notifications > Servers tab)., 015 [-SMTP-5c45c867e4b03b89a49b78ba] WARN Notification|SMTP dispatch failed (Reason: Sending the email to the following server failed : email.server.com:25), 100 [-SMTP-5c45c867e4b03b89a49b78ba] WARN SystemOperation|Failed to forward ResolvedNotification{server=5c45c867e4b03b89a49b78ba, destination=5c45c854e4b03b89a49b78b9, content-length=30681}, , , there is a difference between the default-multi-valued parameter and multi-valued parameter meta key values, and the new Endpoint, UEBA, and Live content rules will not work. The same is true for missing single-valued meta keys. Completing the "Update the Multi-Valued and Single-Valued Parameter Meta Keys for the latest Endpoint, UEBA, and RSA Live Content Rules" procedure in the ESA Configuration Guide should fix the issue., , 602 [ deployment-0] WARN Stream|[alert, alert_id, browserprint, cert_thumbprint, checksum, checksum_all, checksum_dst, checksum_src , 602 [ deployment-0] WARN Stream|[accesses, context_target, file_attributes, logon_type_desc, packets] are still MISSING from single-valued, , using select *), you may see that data privacy (if enabled) and the Pivot to Investigate > Navigate link accessed from a context tooltip in the Respond Incident Details view does not work., you do not need to follow these steps for data privacy, instead, see How to Remove Sensitive Meta Keys Globally from All Alerts. However, you need to follow these steps if you want to enable the Pivot to Investigate > Navigate link accessed from a context tooltip in the Respond Incident Details view. , see Reserved keywords., ESA cannot recognize it and the rule will fail to deploy.

The following figure shows the file configured for the custom meta key "Event Source ID" with index settings of "IndexNone" with a format of "Text”.

To save and deploy the new setting on the NetWitness host, select the Apply button. To force the change, restart the Concentrator service or you can wait until the next polling interval for the change to be recognized.

The XML file can also be deployed to other NetWitness hosts by clicking the Push button and selecting the destination NetWitness host. Only deploy the XML file to a NetWitness host that runs that service (that is, other Concentrators).

Update any ESA rule that selects only certain meta from the session to include the ESA generated event_source_id meta key. Add the event_source_id meta key to the SELECT statement. See the following example rule.

Copy

Example Rule with event_source_id

@RSAAlert
SELECT user_dst, reference_id, hostname, event_source_id FROM
Event(
device_class='Windows Hosts',
reference_id IN ('4624' redeploy the ESA rule deployment that contains the ESA rule., ,  see How ESA Handles Sensitive Data. For information on the stragegy and benefits of obfuscating data, see the Data Privacy Management Guide. , , ,  go to   (Admin) > Hosts. If the host is down, the system parameters will not display (updating host information can sometimes be delayed), the Services display in red, and you may see an error message. , , , ,  contact your NetWitness Administrator to restart it. Otherwise, go to Step 2. , , you can go to Health & Wellness and view the last known metrics to see where potential issues are occurring. The most common problem is that your ESA service is exceeding memory thresholds, which causes it to stop or fail. In NetWitness 11.5 and later, see also View Health Statistics and Trends for ESA Correlation in New Health & Wellness., , , , , , , , enter the following and click Apply:, , , , , , , , , , , , , , you can also see details of your ESA Correlation service performance. 
, , , , and use the following filters to view the following statistics:,  3 days 8 hours 25 minutes 46 seconds
       Correlation ServerProcess jvmMemory Total Max163 GB
       Correlation Server, , , , , , , , , , , , , , , , , , , , , , , continue to step 3. , , select your ESA service, and then select  > Start.

Return to the ESA Service to troubleshoot which rules have created memory issues.

, you may need to call Customer Support to get the services to start., continue to step 4., , you can review the stats for your rules to see which rules are consuming too many resources. Sometimes, ESA services fail because a rule is generating too many alerts or a rule is matching too many events. Check for both of these issues if you have determined that memory usage is causing your ESA service to shut down. , , go to Respond > Alerts. In the Filters panel on the left, in the Alert Names section, select the alert name for the rule. The number of alerts with that name appears at the bottom of the Alerts list results. If the number is significantly high for a particular rule, you need to disable the rule and rewrite it to be more efficient., , , , click Reset., , which can use up excessive memory. This typically occurs if you create a large event window where a great number of events accumulate without triggering an alert. This is a problem because each event is stored in memory while the rule waits for the alert to trigger. To check for this issue, go to (Configure) > ESA Rules > Services. From there, you can see the number of events that were matched in the Events Matched column for the deployment. If a high number of events were matched for a given rule, you can investigate the rule further to see if you can make it more efficient., , , , , disable them and rewrite rules so that they don't generate such a high volume of alerts or events. For pointers on how to write more efficient rules, see Best Practices., , go to (Configure) > ESA Rules > Services, and select the rules you want to disable in the Deployed Rules Stats field.

Select Disable to disable the rules.

, , go to (Configure) > ESA Rules > Rules tab > Rule Library.

For each rule that you repair, do the following:

Select the rule to edit and then select > Edit.
Edit the rule to be more efficient. For instructions on creating rules, see Add Rules to the Rule Library
When you are satisfied with your rule, you can save the rule as a trial rule to ensure that any memory issues do not affect ESA services performance. To do this, follow the steps listed in Working with Trial Rules.

, , see topic Create a Deployment in the Live Services Management Guide., , they should automatically show as enabled. If not, you can enable the rules., and select the ESA service in the options panel.

On the deployment tab for the deployment that contains the rules, in the Deployed Rule Stats section, look at the status of the rules in the Enable column. Enabled rules show a green circle. If the rules show a white circle, you can enable the rules.

To enable rules, select the rules you want to enable and select Enable above the table.

, , check to see if the service is stopping and restarting in a loop. To do this, go to the ESA Correlation logs. You can use SSH to get in the system and go to: /var/log/netwitness/correlation-server/correlation-server.log., , , some settings are no longer available in the user interface. In addition to the standard troubleshooting methods available, you can use the nw-shell utility to perform advanced troubleshooting of the ESA Correlation service and rules. For detailed information on the nw-shell utility, see the NetWitness Shell User Guide., , , , , , , , go to (Admin) > Health & Wellness > System Stats Browser.

In the System Stats Browser, use the following filters and then click Apply.

Host: Select your ESA host.
Component: Select Correlation Server.
Statistic: Type - Name (put a space between the dash and name).

In the Statistic column, locate your deployment followed by - Name. The name in the Value field is YOUR ENGINE NAME.
netwitness_11.5_deployname.png

In the above example, the deployment name is DeploymentA and the engine name is deployment-a-sa-managed.

, , type nw-shell.

Connect to ESA:

Go to your ESA physical host and type the command:
cat /etc/netwitness/correlation-server/service-id
Go back to the NW server and nw-shell and connect to the correlation server:
connect --service correlation-server.ID

Type login.
Enter the username and password of the admin user.

, , , , , , type:
cd /rsa/correlation/engine/execute-query

Type invoke '{"engineName":" ", "query":" "}

Where = the engine name that you located in Find Your Engine Name for Nw-Shell.

Where = the select statement into the named window.

, "query":"select * from UserLoginProfile"}', , , , , , , , , , , , type:
cd /rsa/correlation/service/stats/get-condensed-metrics

Type invoke ' '
Where is the engine name that you located in Find Your Engine Name for Nw-Shell.

, , , , , , , , , , , , , , , , , , , , , , ,