How to fix all disks of DAC that are UGOOD(unconfigured Good) and DGs are frgn(foreign)
Issue
The status of all disks of DAC is UGOOD(Unconfigured good) and DGs are frgn(foreign) state.
In the MegaRAID firmware log, All disks in the DAC were reseted/removed/added. Because all the disks were removed, the VD was also degraded and removed. After that the disks were added, but their state were unconfigured good because the VD was removed.
05/01/20 20:03:19: EVT#17668-05/01/20 20:03:19: 114=State change on PD 0e(e0x0f/s2) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17672-05/01/20 20:03:19: 114=State change on PD 0b(e0x0f/s0) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17677-05/01/20 20:03:19: 114=State change on PD 10(e0x0f/s1) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17680-05/01/20 20:03:19: 114=State change on PD 01(e0x0f/s13) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17683-05/01/20 20:03:19: 114=State change on PD 02(e0x0f/s12) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17686-05/01/20 20:03:19: 114=State change on PD 03(e0x0f/s11) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17687-05/01/20 20:03:19: 114=State change on PD 04(e0x0f/s10) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17688-05/01/20 20:03:19: 114=State change on PD 05(e0x0f/s9) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17689-05/01/20 20:03:19: 114=State change on PD 06(e0x0f/s8) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17690-05/01/20 20:03:19: 114=State change on PD 07(e0x0f/s7) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17691-05/01/20 20:03:19: 114=State change on PD 08(e0x0f/s6) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17692-05/01/20 20:03:19: 114=State change on PD 09(e0x0f/s5) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17693-05/01/20 20:03:19: 114=State change on PD 0a(e0x0f/s4) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17694-05/01/20 20:03:19: 114=State change on PD 0c(e0x0f/s3) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17669-05/01/20 20:03:19: 81=State change on VD 00/0 from OPTIMAL(3) to DEGRADED(2)
05/01/20 20:03:19: EVT#17673-05/01/20 20:03:19: 81=State change on VD 00/0 from DEGRADED(2) to OFFLINE(0)
05/01/20 20:03:20: EVT#17741-05/01/20 20:03:20: 139=Deleted VD 00/0
05/01/20 20:03:19: EVT#17672-05/01/20 20:03:19: 114=State change on PD 0b(e0x0f/s0) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17677-05/01/20 20:03:19: 114=State change on PD 10(e0x0f/s1) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17680-05/01/20 20:03:19: 114=State change on PD 01(e0x0f/s13) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17683-05/01/20 20:03:19: 114=State change on PD 02(e0x0f/s12) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17686-05/01/20 20:03:19: 114=State change on PD 03(e0x0f/s11) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17687-05/01/20 20:03:19: 114=State change on PD 04(e0x0f/s10) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17688-05/01/20 20:03:19: 114=State change on PD 05(e0x0f/s9) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17689-05/01/20 20:03:19: 114=State change on PD 06(e0x0f/s8) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17690-05/01/20 20:03:19: 114=State change on PD 07(e0x0f/s7) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17691-05/01/20 20:03:19: 114=State change on PD 08(e0x0f/s6) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17692-05/01/20 20:03:19: 114=State change on PD 09(e0x0f/s5) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17693-05/01/20 20:03:19: 114=State change on PD 0a(e0x0f/s4) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17694-05/01/20 20:03:19: 114=State change on PD 0c(e0x0f/s3) from ONLINE(18) to FAILED(11)
05/01/20 20:03:19: EVT#17669-05/01/20 20:03:19: 81=State change on VD 00/0 from OPTIMAL(3) to DEGRADED(2)
05/01/20 20:03:19: EVT#17673-05/01/20 20:03:19: 81=State change on VD 00/0 from DEGRADED(2) to OFFLINE(0)
05/01/20 20:03:20: EVT#17741-05/01/20 20:03:20: 139=Deleted VD 00/0
05/01/20 20:03:20: EVT#17726-05/01/20 20:03:20: 114=State change on PD 01(e0x0f/s13) from FAILED(11) to UNCONFIGURED_BAD(1)
05/01/20 20:04:29: EVT#17762-05/01/20 20:04:29: 114=State change on PD 01(e0x0f/s13) from UNCONFIGURED_BAD(1) to UNCONFIGURED_GOOD(0)
05/01/20 20:04:29: EVT#17762-05/01/20 20:04:29: 114=State change on PD 01(e0x0f/s13) from UNCONFIGURED_BAD(1) to UNCONFIGURED_GOOD(0)
Resolution
Cause
Communication failure between Appliance and DAC.Possible root cause
1. Bad cable connection between the RSA NetWitness appliance(Decoder or Concentrator) and DAC.2. Bad RAID card on RSA NetWitness Appliance.
3. Bad Controller on DAC.
Resolution to recover the filesystems
- Stop capture(Decoder) or aggregation(Concentrator)
- Disable Capture autostart(Decoder) or aggregate autostart(Concentrator)
- Connect to the appliance via putty
- Edit /etc/fstab file to comment out /var/netwitness/
filesystems by placing # before each line. Note: <service> will be decoder/logdecoder/concentrator - Reboot the appliance in putty
- Import the foreign disks
# /opt/MegaRAID/MegaCli/MegaCli64 -CfgForeign -Import -a1
- Run nwraidtool.py and check that all disks are online and DG is not a foreign state.
- Remove comments in /etc/fstab added in step 4.
- Mount all file systems.
# mount -a - Restart services.
# systemctl restartNote: <service> will be nwdecoder.service/nwlogdecoder.service/nwconcentrator.service - Start capture/aggregation and enable autostart which disabled in step 2.
Product Details
RSA Product Set: RSA NetWitness Logs & NetworkRSA Product/Service Type: Decoder, Concentrator, Logdecoder
RSA Version/Condition: 11.3.1.0
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue