RAID controller swap procedure for RSA Security Analytics and RSA NetWitness Platform appliances

Issue

RAID controller swap procedure for RSA Security Analytics and RSA NetWitness appliances.
Due to an RMA, I need to swap the RAID controller in my appliance for a new one. What steps do I take to accomplish this?

Resolution

Please follow these steps:

In /etc/fstab, comment out (by adding a # to the beginning of the line) all filesystems that mount under /var/netwitness.
(optional but recommended) Edit /boot/grub/grub.conf and remove the following part of the active boot section’s ‘kernel’ line: ‘console=ttyS0,115200n8r’. This will disable serial console redirection at boot should the appliance go into maintenance mode should it not be able to mount a RAID filesystem. If one does not follow this step and the appliance goes into maintenance mode at boot, you will only be able to see the prompts and interact with the OS by attaching a console device to the serial port rather than by using VGA/Keyboard/Mouse.
Using NetWitness Administrator, stop capture on the Decoder. One should do this before shutting the service down to ensure all indexes are saved properly so as to prevent a need to reindex any sessions upon the next start of the service.
Stop the service with the systemctl stop nwdecoder command.
Run this command to shut down the appliance:
shutdown -h now
Take note of which port the DACs are plugged into, and which internal port the internal RAID enclosure is connected to.
Open the case and perform the swap of the RAID controller, plugging the cables back into their original slots. Move the cable for the old battery backup unit (BBU) from the old to the new controller. Power the appliance back on.
During boot, you should see a message that says ‘foreign configuration detected’ on the RAID controller, or words to that effect. Hit ‘F’ to try to import. It’s alright if you miss this part; the RAID configuration can be imported after the OS boots.
The OS should now fully boot, since you commented out all the hardware RAID file systems in step 1.
Run nwraidutil.pl on the appliance. Check for the presence of all enclosures and DACs. If all RAID disks are reported as being online, skip ahead to step 13. If the RAID disks are in an Unconfigured(Good) state, skip ahead to step 12. If you don't see all of your expected enclosures, check your cabling and ensure all cables are connected to the same ports as before the swap (see step 6).
If the disks are in an Unconfigured(Bad) state, run this command:
Note: For newer version of NetWitness 11.x or 12.x you may need to substitute /opt/MegaRAID/MegaCli/MegaCli64 with /opt/MegaRAID/perccli/perccli64 if the commands return an error.

/opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[ENCLOSURE:DISK,ENCLOSURE:SLOT] -a0 (substitute enclosure and slot number for each drive in an Unconfigured(Bad) state, and substitute correct adapter number (i.e, -a1), where appropriate. If successful, proceed to step 12.

E.g.:
/opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[6:0,6:2,6:3,6:4,6:5,6:6,6:7,6:8,6:9,6:10,6:11,6:11] -a0
/opt/MegaRAID/MegaCli/MegaCli64 PDMakeGood -PhysDrv[25:1,25:2,25:3,25:4,25:5,25:6,25:7,25:8,25:9,25:10,25:11,25:12] -a1

If successful, proceed to step 12. If you cannot get past this step please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d to the case.
If the disks in step are in a foreign state (run nwraidutil.pl again to verify). Run these commands:
/opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -aall

If successful, proceed to step 13. If you cannot get past this step please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d to the case.
Run the following commands:
pvscan
vgscan
lvscan
lvdisplay -C
vgmknodes
If the above commands indicate that physical volumes (PV) and volume groups (VG) are detected but 'lvdisplay -C' indicates the logical volumes are not online (the attributes will be '-wi---' if offline, '-wi-ao' if online), then please run these commands:
lvchange -ay /dev/ /
vgmknodes

E.g.
lvchange -ay /dev/decodersmall/decoroot
lvchange -ay /dev/decodersmall/index
lvchange -ay /dev/decodersmall/metadb
lvchange -ay /dev/decodersmall/sessiondb
lvchange -ay /dev/decoder/packetdb
lvchange -ay /dev/decoder0/packetdb
lvchange -ay /dev/decoder1/packetdb
vgmknodes
If the output of steps 13/14 shows that all LVM RAID volumes are detected and online, you can remove the comments we added in step 1 to the /var/netwitness filesystems in /etc/fstab.
Run the following command:
mount -a

If your filesystems do not mount, please open a case with RSA NetWitness Support for assistance and attach the output of nwtech.sh -d to the case.
If all went well and all of the RAID filesystems mounted, you may start the Decoder process back up with systemctl start nwdecoder and watch /var/log/messages for errors. If the Decoder fails to load, please open a case with RSA NetWitness Support for assistance and attach the *full* output of nwtech.sh to the case.

Internal Comments

UserName:wirthr1
6/21/2012 3:14:03 PM - Solution Number 00000546
Solution Number 00000546

UserName:shurtj
6/4/2014 7:54:25 PM - Changed Article Type
Changed article type from corrective to how-to and modified the statements and title accordingly to abide by Primus best practices.

James Moon -- 27 Aug 2019
Changed Resolution field to use the current commands for SA10.x and NW 11.x. Replaced the special characters with the correct ones(added during the automatic KB conversion)

Product Details

Netwitness Product Set: Netwitness Platform
Netwitness Product/Service Type: Core appliance
Netwitness Version/Condition: 11.x, 12.x or later
Platform: CentOS/Alma Linux

Approval Reviewer Queue

Technical approval queue