Decoder complains about space on packetdb but there is plenty of space and inodes available in RSA NetWitness Logs and Packets

Issue

Decoder stop working because it cannot create the database files on packetdb complain about the space but there is 1.6TB space left on the partition.

 [Utils] [failure] open failed: path: /var/netwitness/decoder/packetdb0/packetdb/packet-000326365.nwpdb flags: 
 
O_RDWR O_CREAT O_TRUNC  [OS Error: (28) - No space left on device]
 
[Utils] [failure] Throw in function int nw::{anonymous}::doOpen(const char*, int)Dynamic exception type: 
 
boost::exception_detail::clone_impl<nw::OsError>std::exception::what: open failed: 
 
path: /var/netwitness/decoder/packetdb0/packetdb/packet-000326365.nwpdb flags: O_RDWR O_CREAT O_TRUNC  
 
[OS Error: (28) - No space left on device][boost::errinfo_at_line_*] = 197
 
Decoder] [info] Capture is stopping

Cause

This is a bug in xfs and xfs_repair couldn’t detect it.

http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_receive_No_space_left_on_device_after_xfs_growfs.3F

Workaround

1. Stop nwdecoder
ex: syststemctl stop nwdecoder
2. Edit /etc/fstab and add inode64 option to packetdb partitions
ex:/dev/decoder0/packetdb /var/netwitness/decoder/packetdb0 xfs noatime,inode64,nosuid 1 2
3. Unmount packet db partitions:
ex: umount /var/netwitness/decoder/packetdb
4. Remount partitions using: mount -a
5. Start nwdecoder, which started capture
ex: systemctl start nwdecoder

Resolution

You must add the "inode64" option to /etc/fstab for Decoder mount points

Product Details

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Decoder
NetWitness Version/Condition: 12.x
Platform: CentOS/Alma

Approval Reviewer Queue

Technical approval queue