Skip to content
  • There are no suggestions because the search field is empty.

RSA NetWitness Reports are failing with max.query.memory limit in Archiver

Issue

NetWitness Reports fail with the following error.

Error occurred while fetching data from source 'ARCHIVER[10.10.10.10]'. Error details : Memory limit of 5.71 GB reached, controlled by setting max.query.memory.

User-added

The reporting-engine log file shows the following error.

grep "Memory limit of" /var/netwitness/re-server/rsa/soc/reporting-engine/logs/reporting-engine.log |tail -2
2021-03-30 04:33:42,723 [EXEC_RULE_33390_20210330043243/Report Name] ERROR com.rsa.soc.datasource.nextgen.nw.service.impl.NwCoreResponseMessage - com.rsa.netwitness.carlos.transport.TransportException: Memory limit of 5.71 GB reached, controlled by setting max.query.memory
com.rsa.soc.datasource.DataSourceException: Error occurred while fetching data from source 'ARCHIVER[10.10.10.10]'. Error details : Memory limit of 5.71 GB reached, controlled by setting max.query.memory.

Cause

This error is due to the default settings in two location of the NetWitness UI, Admin > Services > {Archiver} > Explore page.
Expand + button on the left to go to,
Archiver > Collections > default > sdk > config > max.query.memory
and
sdk > config > max.query.memory

User-added

Workaround

Please increase max.query.memory to higher value than default or, can use 0 as unlimited memory.
Be aware the unlimited memory setting for max.query.memory may fail the Archiver service when queries consume high memory in the Server.

Resolution

Before changing the default max.query.memory setting, try the following.
  1. Review the query in the Report Rules and optimize or simplify.
  2. Reduce the time period that the Report runs for.
 

Calculate a max.query.memory setting


Please use the below notes to set the optimal value using the below points.

Max Query Memory:
The max.query.memory setting limits the amount of RAM used per query. This configuration would stop the queries from running if they require more than the configured value. By default, based on the appliance, it would be around 4-6 GB.

This particular setting should be changed if the queries require more memory for completion. Basically, your query requires more memory than the configured value to populate the results completely. Queries will not return partial results; if they hit the memory limit the report will be auto-canceled by the system.

The max.query.memory is a threshold on the maximum amount of memory for one query. The max.concurrent.queries value is how many query operations are allowed on the database simultaneously. The multiplicative factor of max.query.memory and max.concurrent queries is the overall cap on the amount of memory(RAM) that would be allocated for the given device.

Conditions and Checks:

For example:

• The Total Memory in the Machine is 120 GB.

• Ideal State Memory Usage is X GB (Memory used by OS when services are running and no queries are being made - no reports are running - no investigation is being done). For example, see the "used" memory in the "free -h" output run on the Archiver appliance.

• Maximum Available Memory for the Investigation Operations would be (120{Total Memory} -X) GB.
Now, If we want to increase the max.query.memory from 5.71 to "N" GB, we should ensure that the following condition is met:

"N" * max.concurrent.queries < (120-X-20) GB

Or if we are increasing the max.query.memory, we should check the above condition. If there is a close gap, we should reduce the max.concurrent.queries from 11 to a smaller probably to 6 or 7. If not, we will put the system in danger where there is a possibility of hitting Out-Of-Memory error, eventually failing the service. In addition to this, we do have value calls in place for the cache and hence the additional (-20 GB).

As a general recommendation, we would say to reduce the max.concurrent.queries if we are going to increase the max.query.memory, with the condition max.concurrent.queries * max.query.memory should be always less than (total memory-idle memory-20) GB, or at least less than total memory in the system (swap memory would be used here and sometimes, we might hit OOM). There are other factors like the number of collections on this Archiver. The multiplicative factor should be at least 20-30 GB less than the total RAM for the Archiver to perform aggregation and other Archiver related activities normally.

Restart the Archiver service after making any change to max.query.memory or max.concurrent.queries

Notes

The Archiver appliance can show the current settings from the command line with the following commands.
  1. max.query.memory settings
    For example.
     
    grep max.query.memory /etc/netwitness/ng/NwArchiver.cfg |cut -d\" -f6 |sort -n
    5.71 GB
    5.71 GB
     
  2. max.concurrent.queries
    For example.

    grep max.concurrent.queries /etc/netwitness/ng/NwArchiver.cfg |cut -d\" -f6 |sort -n
    11
    22
     
  3. {Total Memory} -X -20) GB
    For example,

    expr `free -g |grep Mem |awk '{print $4}'` - 20
    92

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Archiver
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7

Summary

This document outlines the procedure to set optimal max.query.memory for completing reports.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue