Skip to content
  • There are no suggestions because the search field is empty.

Unable to start Aggregation due to onboard disk failure in case of S4S Hybrid in RSA NetWitness Platform

Issue

S4S Hybrid's index/warec volume is configured by RAID 0 (no fault tolerance or redundancy).
Therefore if that disk fails, you are not able to start aggregation in SA Web GUI.
You may see the following error on a problematic Hybrid device.
# mount -a
mount: /dev/mapper/VolGroup03-concinde: can't read superblock
mount: /dev/mapper/VolGroup03-warec: can't read superblock
# less -f /var/log/messages | grep aggregation
xxx NwConcentrator[22376]: [Aggregation] [failure] Failed to stop aggregation because writev failed [OS Error: (5) - Input/output error]
xxx NwConcentrator[22376]: [Aggregation] [info] CAS is attempting to restart aggregation from failed state.

Also, you can see the following disk information in this case. (Enclosure: 32, Slot: 9 is missing)
# nwraidutil.pl
RSA NetWitness RAID Utility Version 2012.09.07
Using /opt/MegaRAID/MegaCli/MegaCli64
------------------------
Logical Disk Information
------------------------
Physical Drive State Legend
-------------------------------------------------------------------------------------
B Unconfigured(Bad) O Online
D Dedicated Hotspare and associated virtual drive number R Rebuild
E Hotspare prefers same enclosure S Solid-State Drive
F Foreign U Unconfigured(Good)
G Global hotspare X Offline
I Hotspare is revertible ! Failed
M Missing ? Unknown state
-------------------------------------------------------------------------------------
NOTE: 'E' does not prohibit a hotspare from being used in another enclosure, it is merely a preference
Logical Drive State Legend
------------------------------------------------
D Degraded X Offline
O Optimal ! Failed
P Partially Degraded ? Unknown state
R Rebuild
------------------------------------------------
Adapter: 0 - PERC H710P Mini
Virtual Disk: 0 (O) - Found 6 of 6 {Raid Level 5, 4.545 TB, 128 KB Stripe Size, WriteThrough, ReadAheadNone, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 32 Slot: 0 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6GLSJ
PD: 1 Enclosure: 32 Slot: 1 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6GL3T
PD: 2 Enclosure: 32 Slot: 2 (O) 931.512 GB SEAGATE ST91000640SS AS099XG5PQFG
PD: 3 Enclosure: 32 Slot: 3 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6H0K0
PD: 4 Enclosure: 32 Slot: 4 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6GM48
PD: 5 Enclosure: 32 Slot: 5 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6H0BG
Virtual Disk: 1 (O) - Found 3 of 3 {Raid Level 5, 1.818 TB, 128 KB Stripe Size, WriteThrough, ReadAheadNone, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 32 Slot: 6 (O) 931.512 GB SEAGATE ST1000NX0453 NS02W470ST05
PD: 1 Enclosure: 32 Slot: 7 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6GFGD
PD: 2 Enclosure: 32 Slot: 8 (O) 931.512 GB SEAGATE ST91000640SS AS099XG6H4QY
Adapter: 1 - PERC H810 Adapter
Virtual Disk: 0 (O) - Found 3 of 3 {Raid Level 5, 3.637 TB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 31 Slot: 0 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXWU7K
PD: 1 Enclosure: 31 Slot: 1 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXYPSK
PD: 2 Enclosure: 31 Slot: 2 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKX9M8K
Virtual Disk: 1 (O) - Found 11 of 11 {Raid Level 5, 18.188 TB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 31 Slot: 3 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKWJLHK
PD: 1 Enclosure: 31 Slot: 4 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXJJGK
PD: 2 Enclosure: 31 Slot: 5 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXNLPK
PD: 3 Enclosure: 31 Slot: 6 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXN7SK
PD: 4 Enclosure: 31 Slot: 7 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXJV1K
PD: 5 Enclosure: 31 Slot: 8 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXV52K
PD: 6 Enclosure: 31 Slot: 9 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXV39K
PD: 7 Enclosure: 31 Slot: 10 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXYHMK
PD: 8 Enclosure: 31 Slot: 11 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXV61K
PD: 9 Enclosure: 31 Slot: 12 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXW3NK
PD: 10 Enclosure: 31 Slot: 13 (O) 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXWPNK
No logical disk problems found.
-------------------------
Physical Disk Information
-------------------------
Physical Drive State Legend
-------------------------------------------------------------------------------------
B Unconfigured(Bad) O Online
D Dedicated Hotspare and associated virtual drive number R Rebuild
E Hotspare prefers same enclosure S Solid-State Drive
F Foreign U Unconfigured(Good)
G Global hotspare X Offline
I Hotspare is revertible ! Failed
M Missing ? Unknown state
-------------------------------------------------------------------------------------
NOTE: 'E' does not prohibit a hotspare from being used in another enclosure, it is merely a preference
Adapters found: 2
Adapter 0 (PERC H710P Mini) enclosures found: 1
Adapter 0 (PERC H710P Mini) enclosure 32 slots found: 9
Encl Slot State P.Fail.Count Raw Size Inquiry Data
32 0 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6GLSJ
32 1 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6GL3T
32 2 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG5PQFG
32 3 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6H0K0
32 4 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6GM48
32 5 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6H0BG
32 6 (O) 0 931.512 GB SEAGATE ST1000NX0453 NS02W470ST05
32 7 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6GFGD
32 8 (O) 0 931.512 GB SEAGATE ST91000640SS AS099XG6H4QY
Adapter 1 (PERC H810 Adapter) enclosures found: 1
Adapter 1 (PERC H810 Adapter) enclosure 31 slots found: 15
Encl Slot State P.Fail.Count Raw Size Inquiry Data
31 0 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXWU7K
31 1 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXYPSK
31 2 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKX9M8K
31 3 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKWJLHK
31 4 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXJJGK
31 5 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXNLPK
31 6 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXN7SK
31 7 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXJV1K
31 8 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXV52K
31 9 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXV39K
31 10 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXYHMK
31 11 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXV61K
31 12 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXW3NK
31 13 (O) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXWPNK
31 14 (GEI) 0 1.819 TB HITACHI HUS72302CLAR2000C1D6YFKXWPTK Hotspare Information
No physical disk problems found.
-------------------
Adapter Information
-------------------
Adapters present: 2
Adapter 0:
[Product Name] PERC H710P Mini
[Firmware Version] 3.130.05-1796
[Serial Number] 44N0277
[Memory] 1024MB
[Alarm Device] Absent
[Alarm Enabled] Disabled
[Memory Correctable Errors] 0
[Memory Uncorrectable Errors] 0
[Battery Backup Unit] Present
[Restore HotSpare on Insertion] Disabled
[Use Full Disk Encryption Only] Yes
[Security Keys Assigned] No
[Security Keys Failed] No
[Security Keys Not Backed Up] No
[Virtual Drives] 2
[VD's Degraded] 0
[VD's Offline] 0
[Failed Disks] 0
Adapter 1:
[Product Name] PERC H810 Adapter
[Firmware Version] 3.130.05-1796
[Serial Number] 3C900AY
[Memory] 1024MB
[Alarm Device] Absent
[Alarm Enabled] Disabled
[Memory Correctable Errors] 0
[Memory Uncorrectable Errors] 0
[Battery Backup Unit] Present
[Restore HotSpare on Insertion] Disabled
[Use Full Disk Encryption Only] Yes
[Security Keys Assigned] No
[Security Keys Failed] No
[Security Keys Not Backed Up] No
[Virtual Drives] 2
[VD's Degraded] 0
[VD's Offline] 0
[Failed Disks] 0

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root
20G 6.9G 12G 38% /
tmpfs 48G 0 48G 0% /dev/shm
/dev/sdf1 248M 142M 94M 61% /boot
/dev/mapper/VolGroup00-usrhome
3.9G 113M 3.6G 4% /home
/dev/mapper/VolGroup02-tmp
20G 44M 19G 1% /tmp
/dev/mapper/VolGroup02-varlog
9.8G 2.2G 7.2G 23% /var/log
/dev/mapper/VolGroup01-nwhome
10G 893M 9.2G 9% /var/netwitness
/dev/mapper/VolGroup02-concroot
30G 4.8G 26G 16% /var/netwitness/concentrator
/dev/mapper/VolGroup03-concinde
300G 3.0G 297G 1% /var/netwitness/concentrator/index
/dev/mapper/VolGroup02-concmeta
1.6T 115G 1.5T 8% /var/netwitness/concentrator/metadb
/dev/mapper/VolGroup02-concsess
200G 1.7G 199G 1% /var/netwitness/concentrator/sessiondb
/dev/mapper/VolGroup01-decoroot
30G 25G 5.4G 83% /var/netwitness/decoder
/dev/mapper/VolGroup01-decoinde
30G 131M 30G 1% /var/netwitness/decoder/index
/dev/mapper/VolGroup01-decometa
300G 218G 83G 73% /var/netwitness/decoder/metadb
/dev/mapper/VolGroup01-decopack
4.2T 4.0T 221G 95% /var/netwitness/decoder/packetdb
/dev/mapper/VolGroup01-decosess
10G 3.6G 6.5G 36% /var/netwitness/decoder/sessiondb
/dev/mapper/VolGroup03-warec
400G 33M 400G 1% /var/netwitness/warehouseconnector
/dev/mapper/VolGroup00-vartmp
5.8G 12M 5.5G 1% /var/tmp
/dev/mapper/concentrator0-sessiondb
373G 2.1G 371G 1% /var/netwitness/concentrator/sessiondb0
/dev/mapper/concentrator0-metadb
3.3T 106G 3.2T 4% /var/netwitness/concentrator/metadb0
/dev/mapper/decoder0-packetdb
19T 18T 943G 95% /var/netwitness/decoder/packetdb0

Cause

For some reason, the disk failed and it needs to be replaced via RMA. (Hardware issue)


Resolution

To resolve this issue, you need to replace the faulty disk via RMA and follow the steps after disk replacement.
  1. Recreate VD
    # /opt/MegaRAID/MegaCli/MegaCli64 -CfgLDAdd -R0[32:9] WB ADRA Cached CachedBadBBU -strpsz128 -nolog -a0
    Note: please double-check “disk enclosure and slot number”.
     
  2. Make the volume GPT
    # parted -s /dev/sdo mklabel gpt
     
  3. Recreate LVM Physical Volume
    # pvcreate /dev/sdo    <---- double-check if sdo exists in /dev, if not, then change it.
     
  4. Create LVM Volume Group
    # vgcreate VolGroup03 /dev/sdo
     
  5. Create LVM Logical Volume (LV)
    # lvcreate -L 400G VolGroup03 -n warec /dev/sdo
    # lvcreate -l +100%FREE VolGroup03 -n concinde /dev/sdo
     
  6. Create XFS file system
    # mkfs.xfs -f /dev/mapper/VolGoup03-concinde
    # mkfs.xfs -f /dev/mapper/VolGoup03-warec
     
  7. Check the following folders for mount point    <--- assume there should be the mount point, but just in case double check it.
    - /var/netwitness/concentrator/index
    - /var/netwitness/warehouseconnector
     
  8. Modify /etc/fstab to comment in the line that contains /var/netwitness/concentrator/index, /var/netwitness/warehouseconnector    <--- assume there should be the entry in /etc/fstab but just in case double check it.
     
  9. Mount file systems
    # mount -a
     
  10. Start following services
    # start nwconcentrator
    # start nwappliance
     
  11. Concentrator > Explorer > index and Concentrator > Explorer > database, reflect mount points using "reconfig update" command.
Once completed, you are able to start aggregation in SA Web GUI. (It will take some time to initialize index)

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.6.x
Platform: CentOS

Summary

Onbaord Disk with Raid 0 failed in S4S Hybrid.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue