Skip to content
  • There are no suggestions because the search field is empty.

Netwitness Reporting Engine service is stopping due to all available space being occupied by previous reports

Issue

Netwitness Reporting Engine service is stopping due to all available space being occupied by previous reports.


Cause

The "df -h" command reveals that the volume which contains the  /home/rsasoc/rsa/soc/reporting-engine directory is only 2G instead of 100G, as shown in the example below.

Filesystem            Size  Used Avail Use% Mounted on
...
/dev/mapper/VolGroup00-usrhome
                      2.0G  1.7G  180M  91% /home
...
/dev/mapper/VolGroup02-broker
                      200G  769M  200G   1% /var/netwitness/broker
...
/dev/mapper/VolGroup00-tmp
                      8.2G  4.4G  3.4G  57% /tmp
...

 

The "pvdisplay -C" command also shows 520 GB of un-allocated space in a different VG [VolGroup02], as shown in the example below.

PV         VG         Fmt  Attr PSize   PFree 
/dev/sda1  VolGroup01 lvm2 a--  136.09g  76.09g
/dev/sdb1  VolGroup02 lvm2 a--  930.97g 520.97g
/dev/sdc2  VolGroup00 lvm2 a--   29.56g      0


Resolution

To resolve the issue, follow the steps below.  The workaround will re-create usrhome in VG VolGroup02 to allocate more space to usrhome.

From the output of "df -h", you can see that there is enough free space in /tmp volume to hold current contents the volumes we are working on.

SSH to appliance and run the following:

1. Stop services writing to volume (predominantly Reporting Engine)

#stop rsasoc
#stop jettysrv
 

 2a. Attempt to unmount volume to make sure nothing else is writing to volume

#umount /dev/mapper/VolGroup00-usrhome
 

 2b. If umount fails, need to comment out the following 2 lines in /etc/fstab by editing file and adding # to start of line:

/dev/mapper/VolGroup00-usrhome /home                   ext4    nosuid          1 2

/dev/mapper/VolGroup02-uax /var/lib/netwitness     xfs     nosuid,noatime  1 2

 Will then need to reboot appliance OS to make sure these services are not running (and changing the contents of these volumes).

Once appliance has reloaded can remove # added to the start of the 2 lines in /etc/fstab
 

3. Remount volumes to copy content:

#mount /dev/mapper/VolGroup00-usrhome
#mount /dev/mapper/VolGroup02-uax    [if reboot was needed]
 

4. Create new LVM in free space of PV /dev/sdb1

#lvcreate -L 100G VolGroup02 -n usrhome /dev/sdb1

 You should then be able to see /dev/VolGroup02/usrhome using 'lvscan'

Note: If LV shows as inactive, you will need to make ACTIVE

#lvchange -ay /dev/VolGroup02/usrhome
 

5. Create file system and check for errors

#mkfs.xfs /dev/VolGroup02/usrhome
#xfs_check /dev/VolGroup02/usrhome
 

6. Mount to a temporary location in file system

#mkdir -p /mnt/tmp
#mount -t xfs /dev/VolGroup02/usrhome /mnt/tmp
 

7. Copy existing contents

#rsync -rEAXogt /home/ /mnt/tmp
 

8. Check destination directories are not empty

#ls /mnt/tmp -l
 

9. umount directories once again

#umount -f /dev/mapper/VolGroup00-usrhome
#umount -f /mnt/tmp
 

10. Remove LVM Logical Volume

It's easiest to see full path of LVs using 'lvscan':

Example output:

  ACTIVE            '/dev/VolGroup00/usrhome' [2.00 GiB] inherit

#lvremove -f /dev/VolGroup00/usrhome
 

11. Amend /etc/fstab to change VolGroup referenced and ensure file system is set to xfs rather then ext4:

BEFORE:

/dev/mapper/VolGroup00-usrhome /home                   ext4    nosuid          1 2

AFTER:

/dev/mapper/VolGroup02-usrhome /home                   xfs    nosuid          1 2
 

12. Re-mount home volume

#mount -a
 

13. "df -h" should now show no /dev/mapper/VolGroup00-usrhome but instead:

Filesystem            Size  Used Avail Use% Mounted on

...

/dev/mapper/VolGroup02-usrhome

                      100G  1.7G  98.3G  1.7% /home
 

14. Cleanup

#rmdir /mnt/tmp
 

15. Restart appliance

All services should load normally

 


Internal Comments

UserName:shurtj
8/7/2014 1:24:32 PM - Updated Article
Updated article and made changes to abide by Primus best practices.

Mirna Amir -- Jun 9 2024
Fixed the title

Evan Pols -- 1-10-2025
 Changing 000002600 to Archive, it no longer applies because the reporting engine is under /var/netwitness, which should have plenty of space compared to older version under /home/rsasoc. Recommending  a new article explaining how to adjust the report history retention in the UI config under Config>General> Retain Report History and also where to clean up old formatted reports under /var/netwitness/re-server/rsa/soc/reporting-engine/formattedReports by using a find command to delete reports older than X days https://www.cyberciti.biz/faq/how-to-find-and-delete-directory-recursively-on-linux-or-unix-like-system/ 


Product Details

Netwitness - Reporting Engine
***INTERNAL ONLY***

Approval Reviewer Queue

Technical approval queue