After Upgrading to NetWitness 12.5+ /var/netwitness/source-server/ begins filling up the /var/netwitness/ filesystem

Issue

The Content Library within Centralized Content Management (CCM) retains older versions of content, even though the UI primarily displays the latest version.
A unified approach to removing these older versions is essential, as the /var/lib/netwitness/source-server/resources directory will otherwise continue to grow over time.
The current script is designed to address this issue by deleting older version of contents from both the file system and MongoDB, while selectively skipping certain entries based on predefined criteria.

Cause

The Source Server service, which manages Centralized Content Management's content library, retains copies of historic versions of Live Content. After upgrading to 12.5 or above, the GeoIP Database and Live Feeds are updated several times a week. Previously, this GeoIP DB was only updated with version upgrades. Each update is large, and over time this will begin to fill up /var/netwitness/ on the Admin-Server/Node-Zero. 

This can be confirmed by examining the largest directories under the /var/netwitness/source-server/resources and looking for multi-GB sized directories with the following command:

Workaround

• Download the content-cleanup-files.zip attached to this KB.
• On the Admin-Server/Node-Zero, make the following directory: 
  
  mkdir /var/netwitness/source-server/content-cleanup-files
• Copy the attached zip (via SCP) to  /var/netwitness/source-server/content-cleanup-files  on the Admin-Server/Node-Zero
• On Node-Zero, confirm the matching sha256sum of the zip:
  
  [root@NW-NODE-ZERO content-cleanup-files]# sha256sum content-cleanup-files.zip
1effae003659e144c2b8f21d48f69aa7fa214e7740ad72bce075d5045c673933 content-cleanup-files.zip
• Unzip the contents:
  
  [root@NW-NODE-ZERO content-cleanup-files]# unzip content-cleanup-files.zip
Archive: content-cleanup-files.zip
inflating: run-cleanup.sh
inflating: cleanup.jar
• Make the scrip executable: 
  
  [root@NW-NODE-ZERO content-cleanup-files]# chmod +x ./run-cleanup.sh
• In the same working directory (/var/netwitness/source-server/content-cleanup-files) do a mongoexport and mongodump (note the dump may be a little larger than 1GB) as a backup before execution. Both will prompt for the deploy_admin password:
  
  [root@NW-NODE-ZERO content-cleanup-files]# mongoexport --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db source-server --collection contentLibrary --out contentLibrary.json
2025-07-22T20:54:46.512+0000 WARNING: --sslAllowInvalidCertificates and --sslAllowInvalidHostnames are deprecated, please use --tlsInsecure instead
Enter password for mongo user:



[root@NW-NODE-ZERO content-cleanup-files]# mongodump --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db source-server
2025-07-22T20:55:10.987+0000 WARNING: --sslAllowInvalidCertificates and --sslAllowInvalidHostnames are deprecated, please use --tlsInsecure instead
Enter password for mongo user:
• Run the script, where it will prompt for the deploy_admin password:
  
  [root@NW-NODE-ZERO content-cleanup-files]# ./run-cleanup.sh
Enter MongoDB password:
  
  
  • The output it generates is too large to include in this text example.
• Once complete, you will end up with the following directories. Upon completion, a folder named content-cleanup-script-output will be created in the same directory. This folder contains multiple JSON files that can assist with future debugging:
  
  [root@NEW-NW11-NW-NODE-ZERO content-cleanup-files]# ls -lrth
total 54M
-rwxrwxrwx. 1 root root 431 Jul 9 09:41 run-cleanup.sh
-rw-rw-rw-. 1 root root 29M Jul 14 16:26 cleanup.jar
-rwxr-xr-x. 1 root root 26M Aug 4 12:13 content-cleanup-files.zip
drwxr-xr-x. 2 root root 4.0K Aug 4 12:14 content-cleanup-script-output
[root@NEW-NW11-NW-NODE-ZERO content-cleanup-files]# cd content-cleanup-
content-cleanup-files.zip content-cleanup-script-output/
[root@NEW-NW11-NW-NODE-ZERO content-cleanup-files]# cd content-cleanup-script-output/
[root@NEW-NW11-NW-NODE-ZERO content-cleanup-script-output]# ls -lrth
total 28K
-rw-r--r--. 1 root root 3 Aug 4 12:14 files-to-be-deleted.json
-rw-r--r--. 1 root root 3 Aug 4 12:14 after-file-deletion-success.json
-rw-r--r--. 1 root root 3 Aug 4 12:14 after-file-deletion-failure.json
-rw-r--r--. 1 root root 3 Aug 4 12:14 mongo-documents-to-be-deleted.json
-rw-r--r--. 1 root root 3 Aug 4 12:14 after-file-deletion-remaining-files.json
-rw-r--r--. 1 root root 3 Aug 4 12:14 after-mongo-deletion-success.json
-rw-r--r--. 1 root root 3 Aug 4 12:14 after-mongo-deletion-failure.json
[root@NEW-NW11-NW-NODE-ZERO content-cleanup-script-output]# du -sh *
4.0K after-file-deletion-failure.json
4.0K after-file-deletion-remaining-files.json
4.0K after-file-deletion-success.json
4.0K after-mongo-deletion-failure.json
4.0K after-mongo-deletion-success.json
4.0K files-to-be-deleted.json
4.0K mongo-documents-to-be-deleted.json

• This process should be repeated every couple of weeks or months as necessary until the code fix is implemented with a future upgrade.

Resolution

NetWitness Engineering has released a tool that can be used periodically to purge the older copies of these large historic Live elements, which include:

• Live Feeds
• GeoIP Databases

It is not necessary to retain the above historic copies, and automatic removal will be addressed in a later release. 

In the meantime, please download the attached zip and implement the steps listed in the workaround section below periodically to ensure the /var/netwitness/ volume does not fill up.

Product Details

NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: Admin Server/Node-Zero 
NetWitness Version/Condition: 12.5+
Platform: AlmaLinux

Approval Reviewer Queue

Technical approval queue