Skip to content
  • There are no suggestions because the search field is empty.

NetWitness Integration-Server offline due to OutOfMemoryError

Issue

Integration-server is offline in UI, and logs show below errors.

/var/log/netwitness/integration-server/integration-server.log:
2021-05-27 09:43:49,219 [nchMessageListenerContainer-19] WARN API|Failure /rsa/notification/send [counter=13662 reason=OutOfMemoryError::Java heap space]
2021-05-27 09:43:49,229 [nchMessageListenerContainer-19] ERROR c.r.a.l.e.t.LaunchMessageListenerContainer|Consumer thread error, thread abort.
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:686)
at java.base/java.lang.StringBuffer.append(StringBuffer.java:409)
at java.base/java.io.StringWriter.write(StringWriter.java:99)
at java.base/java.io.Writer.write(Writer.java:213)


 

Cause

Integration-Server service gets out of memory due to high number of email alerts being processed.


Resolution

1. Please login NwServer and backup integration server config file:

cp -av /etc/netwitness/integration-server/integration-server.conf /root/integration-server.conf.bak



2. Edit the file to set the JAVA_MAX_HEAP_GB from 3G to 5G :

vi /etc/netwitness/integration-server/integration-server.conf

Before:

# This is the integration-server daemon configuration file.
#
# This file can be used to tweak the Java VirtualMachine configuration and is *not* the application configuration file.
# For most configuration changes the file you're probably looking for is: /etc/netwitness/integration-server/integration-server.yml

# No /var/log logs required.
LOG_FOLDER=/dev
LOG_FILENAME=null

# CentOS does not have start-stop-daemon
USE_START_STOP_DAEMON=false

# The service will run using the specified Java installation
JAVA_HOME=/etc/alternatives/jre_17

# Any options defined here will be passed to the JVM when it is launched
JAVA_OPTS="-XX:+UseG1GC -Dlog4j2.formatMsgNoLookups=true -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Djava.security.egd=file:/dev/./urandom -Xmx$ {JAVA_MAX_HEAP_GB:-3G}"

After:

# This is the integration-server daemon configuration file.
#
# This file can be used to tweak the Java VirtualMachine configuration and is *not* the application configuration file.
# For most configuration changes the file you're probably looking for is: /etc/netwitness/integration-server/integration-server.yml

# No /var/log logs required.
LOG_FOLDER=/dev
LOG_FILENAME=null

# CentOS does not have start-stop-daemon
USE_START_STOP_DAEMON=false

# The service will run using the specified Java installation
JAVA_HOME=/etc/alternatives/jre_17

# Any options defined here will be passed to the JVM when it is launched
JAVA_OPTS="-XX:+UseG1GC -Dlog4j2.formatMsgNoLookups=true -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Djava.security.egd=file:/dev/./urandom -Xmx$ {JAVA_MAX_HEAP_GB:-5G}"



3. Restart the service using below command.

systemctl restart  rsa-nw-integration-server.service


  4. Check the service status to confirm that the changes have been applied (look for -Xmx5G):

[root@AdminServer ~]# systemctl status rsa-nw-integration-server | grep -i xmx
└─2057576 /etc/alternatives/jre_17/bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -XX:+UseG1GC -Dlog4j2.formatMsgNoLookups=true -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Djava.security.egd=file:/dev/./urandom -Xmx5G -jar /usr/sbin/integration-server.jar --rsa.security.pki.ciphers=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 --rsa.logging.console=false


 


Notes

Should you continue to encounter Out of Memory errors after increasing the heap size to a maximum of 5G, please repeat the process and increment up the Max Heap in 1G increments until you get to a point where the integration server is not running out of memory. 


Product Details

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Admin Server, Integration Server
NetWitness Version/Condition: 11.x, 12.x
Platform: CentOS / AlmaLinux
O/S Version: 7 / 8.9


Summary

This document outlines the procedure to increase memory for integration server.


Approval Reviewer Queue

Technical approval queue