How to replace a single hard drive in an RSA NetWitness Platform appliance

Issue

Resolution

Notes:

• Steps 1 to 7 may be done remotely either via SSH or DRAC Console.
• If you need to replace more than one disk, then replace them in the following order of preference:
  • Disks on-board the appliance head unit (versus a DAC)
  • Disks which currently have physical state of Failed, Offline, Unknown state or Missing
  • Disks which currently have physical state of Unconfigured or Hotspare
• Ensure that if you are replacing the OS drive on an appliance (In Series 4 and Series4S appliances that are NOT Hybrids, AIO or Warehouse Nodes, these show as 136.732 GB or 160 GB in nwraidutil.pl output), that you shutdown the appliance operating system first. Prior to shutting down, it is always recommended to stop capture on network/log decoders and aggregation on concentrators in either the NetWitness UI or REST interface and stop the services.

• Warning: If you need to replace more than one disk, do not replace all disks simultaneously. If the previous disk was a virtual disk/RAID volume member, then wait for any RAID rebuild to complete (see step 6 for the command).
• For SA Warehouse Nodes (MAPR SAW), please refer to the article entitled Security Analytics | How to replace faulty disk on RSA Security Analytics Warehouse (SAW) node in RSA Security Analytics 10.3 and higher instead of this article for disk replacement.

Steps for Replacing Single Disk

1. Confirm the logical location of the disk.

   Use nwraidutil.pl to confirm Adapter and whether disk is on-board the appliance head unit or on a DAC
   nwraidutil.pl - How to Download and Use the RSA NetWitness RAID Utility
   
   Note 1: In RSA NetWitness 11.x, a version of nwraidutil.pl has been included in rsa-sa-tools RPM which installs to /opt/rsa/saTools/nwraidutil.pl. This may not be the newest version of this script.
   Note 2: In RSA NetWitness 11.3 above, nwraidutil.pl has been deprecated by /opt/rsa/saTools/nwraidtool.py which uses perccli instead of MegaCli64. In this case, You can find hard disk drive replacement guide in https://community.rsa.com/docs/DOC-57956.
   
   Record the enclosure number and the slot number of the disk to be replaced if not already known, for example, Enclosure 15 Slot 2 is referred to as [15:2] in commands below.
   Please see Example 2 in the notes below.

   The disks in the appliance head unit are typically shown as either 136.732 GB (146 GB) or 931.512 GB (1 TB) and for Series 4 / S4S Appliances, Adapter 0 is usually one of the models Dell PERC H700/H710.The disks in a JBOD/DAC are typically in enclosures that are attached to Adapter 1. For Series 4 / S4S Appliances, Adapter 1 is usually one of the following models Dell PERC H800/H810.
   In some rare instances, Linux will reverse the Adapters and the onboard disks are on Adapter 1.
    
2. Confirm the current role of the disk - if it is part of a Virtual Disk (VD), a Hotspare or Unconfigured/Failed.
   Use nwraidutil.pl and examine Physical Disk Information.
   1. If disk to be replaced shows as GEI or ID-X, for example, ID-0 in the Physical Disk Information, then it is a hotspare.
   2. If disk to be replaced shows as (U) - Unconfigured, (X) - Offline or (!) - Failed in the Physical Disk Information then it is currently not part of VD or hotspare.
      Please see Example 2 in the notes below.
      
      Use nwraidutil.pl and examine Logical Disk Information.
   3. If disk to be replaced shows as (O) in the Logical Disk Information, then it part of a VD set.
      Please see Example 1 in the notes below.
3. Take Disk Offline
   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 -PDOffline -PhysDrv[ENCLOSURE:SLOT] -a<adapter_num>
   Example - Replacing the 14th disk in 1st DAC
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 -PDOffline -PhysDrv[15:13] -a1
   
   Example Output:
   Adapter: 1: EnclId-15 SlotId-13 is marked Offline.
   
   Exit Code: 0x00
   
    Disk state may change to (U) for Unconfigured.
4. Mark drive as missing [optional]
   Note: This does not need to be done if the disk is not currently part of a VD and so does not show as (O) in Logical Disk Information.
   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdMarkMissing -physdrv[ENCLOSURE:SLOT] -a<adapter_num>
   Example - Replacing the 14th disk in 1st DAC
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdMarkMissing -physdrv[15:13] -a1
   
   Example Output:
   Adapter: 1: EnclId-15 SlotId-13 is marked Missing.
   
   Exit Code: 0x00
   
    
5. Prepare for removal.

   Note: This does not need to be done if the disk is not currently part of a VD and so does not show as (O) in Logical Disk Information.

   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdPrpRmv -physdrv[ENCLOSURE:SLOT] -a<adapter_num>
   Example - Replacing the 14th disk in 1st DAC
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdPrpRmv -physdrv[15:13] -a1
   
   Example Output:
   Prepare for removal Success
   
   Exit Code: 0x00
   
6. Show progress of rebuild [optional].
   If there is a hotspare disk available and a member of the VD is taken offline, then the hotspare disk state will change to R - Rebuild once step 3 is run.
   This is not applicable if disk taken offline was not previously (O) in the Logical Disk Information.
   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv[ENCLOSURE:SLOT] -a<adapter_num>
   Example - Watching rebuild of hotspare to member of VD
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -ShowProg -PhysDrv[15:14] -a1
   
   Example Output:
   Rebuild Progress on Device at Enclosure 15, Slot 14 Completed 3% in 7 Minutes.
   
   Exit Code: 0x00
7. Show the physical location of the drive to be replaced.
   Start Flashing a Drives Amber LED: (you may need to remove DAC faceplate to properly see drive LEDs)
   /opt/MegaRAID/MegaCli/MegaCli64 -PdLocate -start -physdrv[ENCLOSURE:SLOT] -a<adapter_num>
   Stop Flashing a Drives LED:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdLocate -stop -physdrv[ENCLOSURE:SLOT] -a<adapter_num>
8. Physically replace the drive.
9. Get missing drives Array and Row (required for next step) [optional if Step 4 is skipped or fails].

   This is optional if Step4 was skipped or failed but physical disk status shows as (U) - Unconfigured.

   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdGetMissing -a<adapter_num>
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 -PdGetMissing -a1
   
   Example Output 1:
       Adapter 1 - No Missing Drive is Found.
   
   Exit Code: 0x00
   
   Example Output 2:
       Adapter 1 - Missing Physical drives
   
       No. Array Row Size Expected
       1 1 13 139392 MB
   
   Exit Code: 0x00
   

   If the output shows "No Missing Drive is Found" then can skip Step 10.

10. Replace Missing Drive.
    Syntax:
    /opt/MegaRAID/MegaCli/MegaCli64 -PdReplaceMissing -PhysDrv[ENCLOSURE:SLOT] -Array<N> -row<N> -a<adapter_num>
    Note: Arrays start a 0 so first array is Array0
    
    Example Command:
    /opt/MegaRAID/MegaCli/MegaCli64 -PdReplaceMissing -PhysDrv[15:13] -Array1 -row13 -a1
    
    Example Output:
    Adapter: 1: Missing PD at Array 1, Row 13 is replaced.
    
    Exit Code: 0x00
    
11. If disk shows as (X) for Offline in nwraidutil.pl output, change disk back to Online.
    Syntax:
    /opt/MegaRAID/MegaCli/MegaCli64 -PDOnline -PhysDrv[ENCLOSURE:SLOT] -a<adapter_num>
    
    Example Command:
    /opt/MegaRAID/MegaCli/MegaCli64 -PDOnline -PhysDrv[15:13] -a1
    
    Example Output:
    EnclId-15SlotId-13 state changed to OnLine.
    
    Exit Code: 0x00
    
12. 1. Show progress of rebuild (not applicable if replaced disk will become hotspare)
       See Step 6 above for command.
        
       OR
        
    2. If the disk was previously a hotspare (or there is currently no hotspare present in the enclosure), then a hotspare needs to be configured.
       nwraidutil.pl should show Physical Disk state as(U) - Unconfigured.
       Example Output:
       15    14    (U)       0             1.819 TB       HITACHI HUS72302CLAR2000C1D6YGKNKHTK
       
       
       If the disk is onboard the head unit or comes from the 1st DAC on a S4/S4S/SA appliance, then it is usually set as Global hotspare with Enclosure Affinity:
       Syntax:
       /opt/MegaRAID/MegaCli/MegaCli64 pdhsp set enclaffinity physdrv[ENCLOSURE:SLOT] -a<adapter_num>
       After running this command disk status will show as GEI.

       
       For disks from enclosures after the 1st DAC, it is usually recommended to configure hotspare as Virtual Disk dedicated hotspare:

       Syntax:
       /opt/MegaRAID/MegaCli/MegaCli64 pdhsp set dedicated Array<N> physdrv[ENCLOSURE:SLOT] –a<adapter_num>
       Example: Setting 14th SATA disk in the 1st DAC of a packet concentrator as VD dedicated hotspare
       /opt/MegaRAID/MegaCli/MegaCli64 pdhsp set dedicated Array1 physdrv[15:13] -a1
       After running this command disk status will show as ID-X. Above example creates a hotspare for 2nd VD (Array1), and so disk status will show as ID-1.
       
       Notes on setting hotspare:
       • In order for a hotspare to take over the role of a disk in an array, the disk needs to have the same capacity as the disk it is replacing. The impact of this is that a SATA disk cannot be a hotspare for the array of Solid State Disk (SSD) found in concentrator DACs.
       • If you are creating a hotspare which is not the last disk in the DAC, consider adding the parameter -nonRevertible to make the hotspare non-revertible.
       • If a disk in VD is still in (R) - Rebuild state, then setting hotspare for this VD will likely fail while this is still rebuilding.
          
13. If the hotspare was previously a VD dedicated hotspare, then one of the features of this hotspare type is the hotspare is revertible (once bad disk is replaced, the hotspare will go back to being a hotspare). The way this is done is using copyback.

1. Wait for copyback to complete (which will take somewhere in the region of 7 hours)
   This can be monitored through the RAID adapter log:
   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 adpeventlog getsinceshutdown –f <output_filename.log> –a<adapter_num>
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 adpeventlog getsinceshutdown –f raid_events_since_boot.log –a1
   
   See Example 3 in Notes below for an example of raid adapter event log.
   
   Copyback progress can also be monitoring using the following command:
   Syntax:
   /opt/MegaRAID/MegaCli/MegaCli64 pdcpybk showprog physdrv[ENCLOSURE:SLOT] –a<adapter_num>
   Example Command:
   /opt/MegaRAID/MegaCli/MegaCli64 pdcpybk showprog physdrv[15:13] -a1
   Example Output:
   Copyback Progress on Device at Enclosure 15, Slot 13 Completed 5% in 21 minutes
   
   Exit Code: 0x00
   
2. Stop copyback and manually set hotspare using the command in Step 11b.

Notes