After upgrading or installing version NetWitness 12.4 or above, /var/netwitness/decoder or /var/netwitness/logdecoder are consuming 100% of the mounted volume

Issue

NetWitness has introduced a new troubleshooting database intended to gather parser statistics (log or packet). By default, these stats write to the new parsestatdb, location on the decoder root volume, which is usually smaller and intended simply as a holding spot for larger mounts intended for the actual packet/log and meta databases. 

This new subdirectory, named parsestatdb on the decoder/logdecoder root mount will appear as follows.

With  new parsestatdb and the existing statdb , the service default sizes mean that the volume, usually allocated only 10-20 GB, can fill up and cause H&W alarms or the decoder/logdecoder services to stop. 

Cause

The cause of this is that the predefined service limits for the parsestatdb and statdb can often exceed the allocated space on /var/netwitness/decoder or /var/netwitness/logdecoder, which may be allocated as little as 10GB on the Almalinux filesystem. 

Workaround

Follow the steps in the above Resolution section to lower the statdb and parsestatdb sizes so they do not fill the /var/netwitness/decoder or /var/netwitness/logdecoder root directories. 

Resolution

Given that these databases are only intended for troubleshooting, and therefore do not need to contain a large amount of historic statistics, the NetWitness administrator should adjust the allocated space on the respective nwdecoder or nwlogdecoder services in the Explore UI interface with these following steps. Please note that both the new parsestatdb and the existing statdb sizes should be adjusted as follows: 

1. Login to the NW UI
2. Go to Admin>Services
3. Choose the Log Decoder or Decoder service's gear icon, and select "Explore"
   1. For the statdb:
      1. Expand Decoder\sys\config
      2. Find the value titled: "Historical Stats Database Directory (stat.dir)"
         1. Update the value to the lower recommended size of "3 GB"
         2. Example: /var/netwitness/decoder/statdb=3 GB
         3. Click outside the box or press enter to commit 
            
            
   2. For the parsestatdb:
      1. Expand the Decoder\decoder\parsers\config page
      2. Find the value titled "Parser Stat Database Directory (parser.stat.dir)"
         1. Update the value to the lower recommended size of "3 GB"
         2. Example: /var/netwitness/decoder/parsestatdb=3 GB
         3. Click outside the box or press enter to commit 
            
            
            
   3. Afterwards, go in to the Decoder|Logdecoder >System page, and restart the service via the UI (note the "Shutdown Service" button simply restarts the service, it does not shut it down):
      
      
      
   4. Once complete, check to ensure the following:
      1. Execute a "df -h" and ensure it shows /var/netwitness/decoder or /var/netwitness/logdecoder are no longer being utilized at or near 100%
      2. Execute a "du -sh" on the /var/netwitness/ /parsestadb and statdb to ensure they're not exceeding 3GB in side. Examples below:
         
         [root@NET-HYBRID ~]# du -sh /var/netwitness/decoder/* | grep -i statdb
578M /var/netwitness/decoder/parsestatdb
2.9G /var/netwitness/decoder/statdb

[root@LOG-HYBRID ~]# du -sh /var/netwitness/logdecoder/* | grep -i statdb
360M /var/netwitness/logdecoder/parsestatdb
46M /var/netwitness/logdecoder/statdb


Product Details

NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: Packet Decoder, Log Decoder
NetWitness Version/Condition: 12.4 and above
Platform:  AlmaLinux

Summary

After upgrading or installing NetWitness 12.4 or later, the root of the volume for /var/netwitness/decoder (Network Decoders) or /var/netwitness/logdecoder (Log Decoders) begins filling to 100%. In this scenario, you may receive a H&W alarm or the decoder or logdecoder services may stop. Often times this is confused with the captured data subdirectories filling up (such as packetdb or metadb), but when running a df -h to examine the filesystem via the CLI, it is observed that those directories, contained on their own mount points, are still being maintained fine and not exceeding the pre-defined limit of 95%. The directory(s) that are now filling up this volume are likely comprised of the statdb (pre-existing) and the new parsestatdb (new as of 12.4).

Approval Reviewer Queue

Technical approval queue