How to troubleshoot a full root partition on RSA Security Analytics appliances

Issue

The root ( / ) partition may become 100% full on a Security Analytics appliance for a variety of reasons.

Resolution

The script below looks only in the root ( / ) partition (-xdev option) and at files larger than 10 MB (this can be changed), and will help identify which file(s) could be taking up the disk space.

[root@ESA-Server ~]# find / -xdev -type f -size +10M -printf '%s %f\n' | sort -n ; for i in $(find / -maxdepth 1 $ ! -name proc $ -type d) ; do echo -n $i": " ; ( find $i -type f | wc -l ) 2>/dev/null ; done | sort -k2 -n
18519885 initramfs-2.6.32-358.18.1.el6.x86_64.img
19334799 initramfs-2.6.32-504.1.3.el6.x86_64.img
19432098 initramfs-2.6.32-431.23.3.el6.x86_64.img
/lost+found: 0
/media: 0
/mnt: 0
/srv: 0
/home: 10
/temp: 21
/boot: 36
/dev: 49
/bin: 83
/sbin: 172
/root: 193
/lib64: 384
/tmp: 854
/etc: 1013
/selinux: 1437
/opt: 1779
/sys: 7140
/lib: 7287
/var: 18816
/usr: 42310
/: 115063

Parameters:
+10M (Files over 10 MB)

Notes

It's uncommon to have files in folders that are used to mount external filesystems (example NFS in warehouse connector), to check this kind of folder without unmounting them use:

mkdir /newroot && mount --bind / /newroot

Inside /newroot you will be able to check inside the folders that are used as mount point.

Scenario:

- Logdecoder that has a warehouse connector using /saw as NFS mounting point.
- NFS communication timeouts, then WHC will write files in /saw (but now is LOCAL FOLDER!!!).
- After mounting again the NFS the files inside /saw will be shadowed.

Product Details

RSA Product Set: Security Analytics
Platform: CentOS
O/S Version: EL5, EL6

Summary

This article provides information on how to find large files on the command line which may assist in troubleshooting file system full situations on Security Analytics devices, which are Linux based appliances.

Approval Reviewer Queue

ASOC Approval Group