RSA NetWitness Logs & Network Health & Wellness indicates /var/netwitness partition is at 100% utilization

Issue

NetWitness Health & Wellness indicates the /var/netwitness partition is at 100% utilization, but you cannot identify the files that are taking up the space on the partition.  

It is possible that a core service may start before a mount point has mounted.  

This can create directories and files on a filesystem other than what is intended, and it can result in the filesystem filling and reporting as full in Health & Wellness and from the Linux command line.

You may see something like the following:
 
In this example the /var/netwitness filesystem appears to be full, but the NetWitness Decoder service continues to run and the /var/netwitness/decoder/packetdb filesystem is less than 60% full.

When examining the contents under the /var/netwitness mount something like the following shows:
 
In other words, no files are observed under the /var/netwitness mount that could be consuming nearly 30GB of disk space.

Cause

In certain circumstances, a service may start capturing or aggregating data before the specified filesystem is mounted.

When this happens, files may be stored under the /var/netwitness mount instead of the filesystem that would normally be mounted in this mount point.

Then, at a later time, when you mount a filesystem on a directory /mount-point, you can no longer access or view the files under the /mount-point directly. They still exist, but the /mount-point now refers to the root of the mounted filesystem, not to the directory that served as a mount point, so the contents of this directory cannot be accessed, at least in this way.

This effectively "hides" or "loses" the files that were previously and erroneously created directly under the filesystem, and makes the filesystem full even though you can't see the files. 

Resolution

To reveal the hidden files, stop the NetWitness services such as nwapliance and nwdecoder then unmount all decoder filesystems using these commands: 
 
Change directories to the /var/netwitness folder and look for any files that remain.  These files are almost certainly unwanted files that were accidentally created at some point in time and should be the files consuming the space on the filesystem.

Once you have removed the unwanted files, mount all filesystems and start all services again using these commands:
 

Notes

In most cases, you will not need to retain the files found in /var/netwitness, but you may wish to copy the files off to an unused filesystem for further evaluation.  Often, the /var/netwitness/warehouseconnector will be unused and have a large amount of unused space that can be used for this purpose.

Though if it was actual Decoder packet or metadata files, then the files could be moved to proper mount points, making sure not to overwrite any existing file names.

Product Details

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: NetWitness Core Appliance
RSA Version/Condition: 10.6.x, 11.x
Platform: CentOS
O/S Version: 6, 7

Summary

How to identify and remove hidden or lost files on the /var/netwitness partition?

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue