Skip to content
  • There are no suggestions because the search field is empty.

After upgrading to 12.4+, /tmp is filling up the root filesystem with tomcat.7050* files on Endpoint Servers

Issue

NetWitness Endpoint Servers utilize /tmp as a temporary holding place for in-progress file download chunks from Endpoint Agents. By default, /tmp resides on the root ( / ) volume, which only contains 30GB space on physical hosts. In larger deployments, this space may not be enough to hold these transient files. The result of this volume filling up can cause the server to do a hard crash. 


Workaround

  • Stop the Endpoint-server:
    systemctl stop rsa-nw-endpoint-server
  • Purge the existing transient files to clean up /tmp:
    rm -rf /tmp/tomcat.7050*
  • Take a backup of the current endpoint-server configuration file:
    cp /etc/netwitness/endpoint-server/endpoint-server.conf /etc/netwitness/endpoint-server/endpoint-server.conf.bak
  • Create the new working directory at /var/netwitness/endpoint-server/temp:
    mkdir -p /var/netwitness/endpoint-server/temp
  • Edit the /etc/netwitness/endpoint-server/endpoint-server.conf file:
    vi /etc/netwitness/endpoint-server/endpoint-server.conf

    • Add this entry after the pkcs12.legacy portion and ensure you don't remove the closing double-quotes:
      -Dserver.tomcat.basedir=/var/netwitness/endpoint-server/temp
    • After the addition of the above line, the file will look exactly like this: 
      # This is the endpoint-server daemon configuration file.
      #
      # This file can be used to tweak the Java VirtualMachine configuration and is *not* the application configuration file.
      # For most configuration changes the file you're probably looking for is: /etc/netwitness/endpoint-server/endpoint-server.yml

      # No /var/log logs required.
      LOG_FOLDER=/dev
      LOG_FILENAME=null

      # CentOS does not have start-stop-daemon
      USE_START_STOP_DAEMON=false

      # The service will run using the specified Java installation
      JAVA_HOME=/etc/alternatives/jre_17

      # Any options defined here will be passed to the JVM when it is launched
      JAVA_OPTS="-XX:+UseG1GC -Dlog4j2.formatMsgNoLookups=true -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Djava.security.egd=file:/dev/./urandom -Xmx${JAVA_MAX_HEAP_GB:-16G} -Dkeystore.pkcs12.legacy -Dserver.tomcat.basedir=/var/netwitness/endpoint-server/temp"

      IMPORTANT NOTE:  If you have previously changed other values in this file, like "JAVA_MAX_HEAP_GB", keep those settings as-is. The above example is from a default configuration file with only the -Dserver.tomcat.basedir=/var/netwitness/endpoint-server/temp added.

  • Run the following command, which will reload the conf file, start the service and watch the new directory we've defined to ensure files are being written. NOTE: It may take 3-5 minutes for the service to start utilizing the new directory after startup:
    systemctl daemon-reload ; systemctl restart rsa-nw-endpoint-server; watch "du -sh /var/netwitness/endpoint-server/temp/work/Tomcat/localhost/ROOT"
  • After running for some time, you can verify everything with this command:
    systemctl status rsa-nw-endpoint-server -l | grep temp ; systemctl status rsa-nw-endpoint-server | grep -i xmx; du -sh /tmp/* | grep -i tomcat| sort -h | tail -n5 ; ls -lrth /tmp | tail -n5 ; df -h | grep "/dev/mapper/netwitness_vg00-root" ; du -sh /var/netwitness/endpoint-server/temp/work/Tomcat/localhost/ROOT ; date

    • This will show us, in order:
      • status of the endpoint-server highlighting the temp redirect directory
      • status of the endpoint-server highlighting the heap size that we didn't alter, but just to confirm
      • the largest 5 files/directories currently in the old /tmp directory containing the string "tomcat" (ignore the docbase ones)
      • The 5 most recently modified files in the old /tmp directory. As long as we don't have tomcat.7050* files (ignore the docbase ones) piling up in there since the last restart, we should be good
      • the current size of the root ( / ) directory, which contains the /tmp
      • the current size of the new redirect directory : /var/netwitness/endpoint-server/temp/work/Tomcat/localhost/ROOT
    • Example Output:
      ├─1871081 /etc/alternatives/jre_17/bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -XX:+UseG1GC -Dlog4j2.formatMsgNoLookups=true -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Djava.security.egd=file:/dev/./urandom -Xmx16G -Dkeystore.pkcs12.legacy -Dserver.tomcat.basedir=/var/netwitness/endpoint-server/temp -jar /usr/sbin/endpoint-server.jar --rsa.security.pki.ciphers=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 --rsa.logging.console=false
      0 /tmp/tomcat-docbase.7050.9581836940981079682
      32K /tmp/hsperfdata_logstash
      32K /tmp/hsperfdata_netwitness
      32K /tmp/hsperfdata_nw-node-infra-server
      2.1M /tmp/endpoint-server.jar-spring-boot-libs-8156b1a8-5692-4cd1-9606-fecdccb25919
      drwx------. 3 netwitness netwitness 18 Jan 16 05:00 tomcat.7050.16951979282981345571
      drwx------. 3 netwitness netwitness 18 Jan 17 05:01 tomcat.7050.5286610209743854783
      drwxr-xr-x. 2 netwitness netwitness 21 Jan 17 17:37 hsperfdata_netwitness
      drwx------. 2 netwitness netwitness 62 Jan 17 17:37 endpoint-server.jar-spring-boot-libs-8156b1a8-5692-4cd1-9606-fecdccb25919
      drwx------. 2 netwitness netwitness 6 Jan 17 17:37 tomcat-docbase.7050.11503779293414689355
      /dev/mapper/netwitness_vg00-root 30G 8.7G 22G 29% /
      454M /var/netwitness/endpoint-server/temp/work/Tomcat/localhost/ROOT
      Fri Jan 17 18:31:34 UTC 2025

NOTE: Until there is a resolution to this in the form of  code update, the steps to edit the /etc/netwitness/endpoint-server/endpoint-server.conf and add the -Dserver.tomcat.basedir=/var/netwitness/endpoint-server/temp  option must be repeated after EVERY upgrade or Hotfix installation. Upgrades and hotfix installations will revert the endpoint-server.conf to defaults.


Resolution

The resolution for this is to change the working /tmp volume to use a mount with more free space. Which is done by:

  • Creating a new subdirectory on /var/netwitness/endpoint-server/temp , which is a much larger volume by default
  • Adding options to the /etc/netwitness/endpoint-server/endpoint-server.conf file to force the endpoint-server service to utilize this new working temp directory.

Detailed steps can be found below in the Workaround section. 


Notes

Should things not continue to work properly and you need to revert, please follow these steps: 

  • systemctl stop rsa-nw-endpoint-server
  • cp /etc/netwitness/endpoint-server/endpoint-server.conf.bak  /etc/netwitness/endpoint-server/endpoint-server.conf
    •  I tried to name the backups consistently, but please ensure you're copying the correct one.
  • systemctl daemon-reload
  • systemctl start rsa-nw-endpoint-server
  • systemctl status -rsa-nw-endpoint-server -l | grep -i temp
    • If nothing appears, then the temp redirect should be gone from the loaded service 

If the new directory is also filling up and running out of space, cronjobs can be added to clean up these temp files after X minutes. Examples of cronjobs:

  • to cleanup /tmp (the default location) and delete all files older than 10 minutes, running once a minute:
    * * * * * find /tmp/tomcat.7050.*/work/Tomcat/localhost/ROOT/ -maxdepth 1 -type f -name "*.tmp" -mmin +10 -delete
  • to cleanup the newly defined temp location (/var/netwitness/endpoint-server/temp/)  and delete all files older than 60 minutes, running once a minute:
    * * * * * find /var/netwitness/endpoint-server/temp/work/Tomcat/localhost/ROOT -maxdepth 1 -type f -name "*.tmp" -mmin +60 -delete

Product Details

NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: Endpoint Server hosts (including Hybrids and Stanadlones)
NetWitness Version/Condition: 12.4, 12.5, 12.6
Platform: AlmaLinux


Summary

After upgrading NetWitness Endpoint Servers to 12.4 or above, the root ( / ) filesystem begins to fill up to 100% and crashes the server. Investigating it further shows that the /tmp subdirectory, which is mounted on root ( / ) , is the culprit. The majority of the files consuming storage in /tmp/ are subdirectories that resemble /tmp/tomcat.7050.13408972331835891568 .


Approval Reviewer Queue

Technical approval queue