How to manually remove a JBOD/DAC that has been added to an RSA NetWitness Logs & Network appliance

Issue

How to manually remove a JBOD/DAC that has been added to an RSA NetWitness Logs & Network/RSA Security Analytics appliance.

On running NwArrayConfig.py you may receive the following message: So the second common use of this KB is removing incomplete DAC addition.

Resolution

Certain situations may require the manual removal of a JBOD or DAC that is attached to an RSA Security Analytics appliance, such as physically moving a DAC or due to an incomplete run of either the nwinitarray.py (deprecated), nwmakearray.py (deprecated), arrayCfg (deprecated) or NwArrayConfig.py script package.

Note: CentOS6 commands should be used for: RSA Security Analytics/NetWitness 10.x
Note: CentOS7 commands should be used for: RSA NetWitness 11.x

Follow the instructions below to perform the procedure.  Gather the supporting information below prior to starting the procedure.

Removing and Re-adding DAC to Log Hybrid
1) Stop services
CentOS6:

CentOS6 10.4.x - 10.6.x Only CentOS7 We need to ensure that nothing is read/writing to these volumes, so will stop volumes from auto-mounting and reboot OS.

2) Comment out the following lines in /etc/fstab relating to the logdecoder and concentrator service by adding # to start of the line:
/etc/fstab example for CentOS6 Log Hybrid:

/etc/fstab example for CentOS7 Log Hybrid:

3) Reboot appliance Note: As volumes are commented out, services will fail to start on restarting OS (which is desirable as we want nothing to use the volumes we are manipulating)

4) Remove # from start of lines in /etc/fstab that were added in step 2.

Steps 5 - 7 are optional (if backup of data is required)
5) Mount volumes:

6) Copy data from appliance (scp perhaps?) for the following locations:  (entirely optional, if locations are empty or if data loss is acceptable can skip)

/var/netwitness/concentrator/sessiondb0
/var/netwitness/concentrator/metadb0
/var/netwitness/logdecoder/packetdb0

7) umount volumes

8) Make LVM Logical Volumes inactive (confirm by lvscan/lsblk -i whether correct LVs) 9) Remove inactive LVM (uses same LVs as lvchange) 10) Remove empty VG (confirm by vgdisplay -C whether correct VGs) 11) Remove PV (confirm block device names by pvscan / lsblk -i) 12) Check that Adapter 1 is the Dell PERC H8XX which communicates with DAC

Example Output (confirms that adapters not reversed and so following commands will use -a1 versus -a0)

13) Remove VDs using MegaCli commands (Note: output of step 12 to confirm whether -a1 or -a0): A more powerful command to delete all VDs on Adapter 1 (potentially more dangerous): Note: If all disks in nwraidutil.pl on DAC have the status of F for foreign config, this may be as simple as clearing the foreign config:
14) Remove any remaining hotspares (as seen in nwraidutil.pl)

If nwraidutil.pl shows [15:14] as hotspare (either GEI or ID-0) then need to remove using MegaCli command:

15) Restart LVM2 daemon
CentOS6:
If daemon is stopped, skip to next step, for example Otherwise CentOS7:
16) Remove the following 3 lines from /etc/fstab
17) Update Log Decoder service configuration (alternative to removing from service in explore mode under database/config)
Edit /etc/netwitness/ng/NwLogdecoder.cfg

Line 19: (searching for packet.dir)
BEFORE:

AFTER:  

18) Update Concentrator service configuration (alternative to removing from service in explore mode under database/config)
Edit /etc/netwitness/ng/NwConcentrator.cfg
Line 60: (searching for meta.dir)
BEFORE:

AFTER:  

Line 75: (searching for session.dir)
BEFORE:

AFTER:  

Note: If service has not been licensed before then cfg file may not be present in /etc/netwitness/ng

19) Re-start services (except nwappliance)
CentOS6

CentOS7: Wait until services look normal in Administration \ Services or Administration \ Devices in RSA NetWitness/SA UI

20) Remove File System Monitors
20a) Stop nwappliance service

CentOS6

CentOS7 20b) Edit NwAppliance service config Check for file system monitors and delete the lines refering to removed LVs:

BEFORE:

AFTER: 20c) Restart the nwappliance service

CentOS6

CentOS7
21) CentOS6 NetWitness/Security Analytics 10.4.x - 10.6.x Only  

22) Re-add DAC
Check using nwraidutil.pl that all disks in DAC show as 'U' for Unconfigured

Security Analytics versions prior to 10.6.X (using arrayCfg script)

Note: For packet hybrids, you may alter number of drives allocated to concentrator service from 7 to something else (defaults to 3 if nothing is entered)

NetWitness 10.6.X

Product Documentation Reference: S4 S5 15-Drive DAC Setup Guide - https://community.rsa.com/docs/DOC-44957
Note: For 10.6, may need to make sure rsa-sa-tools RPM is the latest as per this reference.

NetWitness 11.X

Product Documentation Reference: S4 S5 15-Drive DAC Setup Guide - https://community.rsa.com/docs/DOC-44957

Notes

General order of operations in adding JBOD/DAC: (as performed by above scripts)
1. Create Hardware RAID Virtual Disk (VD) using MegaCli commands - as seen in nwraidutil.pl
You can see the VD presented as a block device to the CentOS6 operating system using: lsblk -i
2. Create partition on block device (optional) - as seen in fdisk -l
3. Create LVM Physical Volume (PV) - as seen in pvscan/pvdisplay -C
If Step 2 was done and block device created in previous step was /dev/sde1 then this step will create PV on /dev/sde1
4. Create LVM Volume Group (VG) - as seen in vgscan/vgdisplay -C
5. Create LVM Logical Volume (LV) - as seen in lvscan/lvdisplay -C
6. Update /etc/fstab
7. Update service configuration e.g. adding new packet.dir location to /etc/netwitness/ng/NwLogdecoder.cfg
8. Add File System Monitors to NwAppliance configuration - /etc/netwitness/ng/NwAppliance.cfg

To remove we go through the list and determine how far through the process it got and work our way backwards
For steps 6 - 8, we'll remove changes last (you could well do them first to reverse Steps 8 -> Step 1 in order)
We can ignore Step2 when removing (could perform something like: fdisk /dev/sde and then d and w to write file table and quit) but not necessary

Supporting Detail:
Using a Log Hybrid as the example:

We can easily tell in this case that the DAC hosts the following volumes:
/var/netwitness/concentrator/sessiondb0
/var/netwitness/concentrator/metadb0
/var/netwitness/logdecoder/packetdb0
 

 

nwraidutil.pl's Physical Disk Information for Adapter 1: