Unable to start concentrator service throwing core files in the RSA NetWitness Platform

Issue

Concentrator service is unable to start throwing core files under /var/netwitness/concentrator/metadb.

Tried to move the core files from the metadb folder and start concentrator service but it doesn't work with new core files. 

From /var/log/messages, you may observe the following messages "nwconcentrator process ended" after the index warning, [Index] [warning] Key ipv6.dst is missing from the most recent slice of 4800 …

NwConcentrator[20178]: [Index] [info] Indexes are being initialized
NwConcentrator[20178]: [Index] [info] Checking integrity of slice 4800
NwConcentrator[20178]: [Network] [info] Accepting connection from trusted peer 127.0.0.1 with subject name CN = 2851ae89-7577-41fe-bfba-1cd7346c2cff
NwConcentrator[20178]: [Engine] [audit] User admin (session 421, 127.0.0.1:49868) has logged in
NwConcentrator[20178]: [Index] [warning] Key ipv6.dst is missing from the most recent slice of 4800, but was found in slice 4799.  This is most likely caused by an index configuration change.
init: nwconcentrator main process (20178) killed by ABRT signal
init: nwconcentrator main process ended, respawning
NwConcentrator[7618]: [meta] [warning] There are core files taking up 41.46 GB on the partition /var/netwitness/concentrator/metadb. Please open a support ticket to troubleshoot.

Cause

For some reason, index slice was broken or corrupted due to unexpected service stop or Filesystem issue.

Workaround

You can try those steps as a workaround.

1. Login to the Concentrator device as a root.
2. Find index slice folder(/var/netwitness/concentrator/index) which is broken. (In this case, /var/netwitness/concentrator/index/managed-values-4800)
3. Make a copy of this folder as a backup and remove this folder.
4. Start concentrator service again as concentrator is able to initialize the index slices even with missing keys.

After a period of time for index initializing, you are able to start concentrator service without any issue.

Note: If this workaround doesn't work, please contact customer support with core files and stack trace output.
https://community.rsa.com/docs/DOC-45444

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.6.x
Platform: CentOS
O/S Version: 6

Summary

Concentrator service is unable to start throwing core files under /var/netwitness/concentrator/metadb.

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue