Customer DAC has Unconfigured Good drive that is outside of any RAID Group and Logical group in RSA Security Analytics

Issue

Single disk in DAC has the status of U for Unconfigured Good in nwraidutil.pl output. There is already a Hotspare present in the DAC of the same disk type.

Below example is a Decoder DAC with 2.5 TB drives in a 32 TB DAC.
nwraidutil.pl output (Physical Disk Information)

Cause

Workaround

The following set of steps is for a packet decoder. If steps are needed for a disk in 2nd DAC or for a Log Decoder or Concentrator then modifications need to be made.

The below instructions are split into 3 sections:

• Preparation
• Increasing the Size of the Volume
• Reconfiguring the Service to Use the New Volume Size

Preparation:

1. Stop Packet capture from SA UI > Decoder > System page
2. SSH to Packet Decoder and run the commands below.

   
   If running Security Analytics/NetWitness for Log and Packets >= 10.4.0 then need to stop puppet service to avoid it restarting nwdecoder while performing the rest of the steps:

   # service puppet stop
   Stop packet decoder service:
   # stop nwdecoder
   If Security Analytics/NetWitness for Log and Packets >= 10.4.0 can also stop the following 2 services to avoid error messages in logs while performing the procedure:
   # stop nwappliance
   # service collectd stop
3. Unmount volume to avoid changes while performing below procedure:
   # umount /var/netwitness/decoder/packetdb
   If umount fails, then comment out the following line by adding # to start of the line in /etc/fstab and reboot the operating system
   /dev/mapper/decoder-packetdb on /var/netwitness/decoder/packetdb type xfs (rw,nosuid,noatime)

1. Note size of /var/netwitness/decoder/packetdb (should be 25T)
   # df -Ph /var/netwitness/decoder/packetdb
   
   Example Output:
   /dev/mapper/decoder-packetdb 25T 8.4T 17T 35% /var/netwitness/decoder/packetdb
2. Add the missing drive to the correct Virtual Disk (VD 1) in Adapter 1.
   # /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -Start -R5 -Add -physdrv[15:9] -L1 -a1
   
   Where LDRecon is the command used to reconstruct the virtual RAID, R5 is the RAID type, L1 is the logical volume, a1 is the adapter number.
   Note: The information provided is for example purposes only.
    
3. Show the progress of the drive add.  You may not see the drive as part of the VD until progress has reached 100% complete.
   # /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -ShowProg -L1 -a1
4. Additional command to view all 11 drives appear.  Also, use this command to verify you have the right VD.  
   # /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -L1 -a1
5. Perform a 'pvresize' to increase size of LVM Physical Volume
   # pvresize /dev/sdd
6. Do a pvdisplay to see the addition of the free available space (column called PFree).  Check /dev/sdd in the output.
   # pvdisplay -C
   
   Example Output:
   PV VG Fmt Attr PSize PFree DevSize PV UUID
   /dev/sda2 VolGroup00 lvm2 a--u 930.47g 0 930.49g VLpHya-bKlX-c4su-Gn9L-QnGZ-RH4S-u47EQ2
   /dev/sdb1 VolGroup01 lvm2 a--u 930.97g 528.97g 931.00g hd3YuD-KPRY-O59l-hQ6v-ItQT-cd1K-yAYith
   /dev/sdc decodersmall lvm2 a--u 5.46t 0 5.46t HaBXxt-lgVG-Rgtw-rxkX-0K8D-npcT-PFNQ2s
   /dev/sdd decoder lvm2 a--u 24.56t 2.73t 24.56t oOjYv6-Lu3R-9766-Y2Mx-2r1N-f92w-6mTg1J
   /dev/sde decoder0 lvm2 a--u 23.64t 0 23.65t UF0GRn-NyxV-iVkt-YPYY-qIcY-0mv2-bGek4o
7. Make LV inactive (precaution only against shared use if volume is already unmounted)
   # lvchange -an /dev/decoder/packetdb
8. Now perform an lvextend command in order to resize the LVM logical volume
   # lvextend -l +100%FREE /dev/decoder/packetdb
   
   Example Output:
   Size of logical volume decoder/packetdb changed from 24.56 TiB (642802 extents) to 27.29 TiB (714254 extents)
   Logical volume packetdb successfully resized
   
   Note: /dev/sdd is used by VG decoder. If you cannot resize, please check that you have the correct volume name.
    
9. Make LV active again and check with pvscan that it shows as ACTIVE
   # lvchange -ay /dev/decoder/packetdb
   # pvscan | grep /dev/decoder/packetdb
   Example output:
   ACTIVE '/dev/decoder/packetdb' [27.28 TiB] inherit
10. Remount the volume (in preparation for xfs_growfs)
    mount /dev/decoder/packetdb
11. Use xfs_growfs command on mounted volume to grow the filesystem to new size
    # xfs_growfs /var/netwitness/decoder/packetdb
12. Check integrity of XFS file system
    # xfs_check /dev/decoder/packetdb
    If you see any issues you may need to run xfs_repair
13. Check new size of /var/netwitness/decoder/packetdb
    # df -Ph /var/netwitness/decoder/packetdb
    
    Example Output:
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/decoder-packetdb 27T 8.4T 18.6T 31% /var/netwitness/decoder/packetdb
14. Restart services
    # service puppet start
    # start nwdecoder

1. Once service is sufficiently initialized, run database reconfig to calculate the new packetdb size in explorer view:
   - SA UI > Services > Packet Decoder > Explore
   - Right-click on the database and select 'Properties'
   - Run the 'reconfig' utility, so that the correct size can be updated
   a. In the drop-down, select reconfig and enter update=false (or update=0) in the Parameters field and hit the Send button.
   
   
     b. Verify the new size of packet.dir (calculated as 95% of volume size seen in `df -hP` or the slightly more accurate `lvdisplay -C`)
     c. Expand out database/config and in right-hand pane double click on packet.dir and update with new value from the previous command output of 1b
    
2. If packet.dir shows as red, then you will need to restart the decoder service in order to reload service with new setting. Stop decoder capture and restart service if necessary.

Notes

Product Details

Summary

A customer suspected that the drive was failed but in fact it was never added as part of the NwArrayConfig script. Also it should be noted that the hotspare is in the wrong location.

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue