RSA NetWitness Logs & Network Log Decoder Hybrid continuously crashing and creating core dump files

Issue

The NetWitness Log Decoder service will not remain started, it keeps looping to start and then crashes and creates core dump files.
This results in filling up to 100% of the filesystem /var/netwitness/logdecoder. Even if you try removing the core files and restarting, the log decoder service will not start up, it just keeps core dumping.

Note the error below repeating in /var/log/messages file while the Log Decoder service tries to start up, until core dump files fill up the filesystem:

 Mar 10 11:31:18 axinsadec3 nw[8057]: [Engine] [warning] Module logdecoder failed to load: Invalid deserialized string length 1868955648. Maximum size exceeded.
 
Mar 10 11:31:18 axinsadec3 nw[8057]: [Engine] [warning] Module logdecoder failed to load: Diagnostic information: Throw in function static void nw::serialization::Serializer<A, std::basic_string<char> >::load(A&, std::string&, unsigned int) [with A = nw::InputArchive; std::string = std::basic_string<char>]Dynamic exception type: N5boost16exception_detail10clone_implIN2nw18SerializationErrorEEEstd::exception::what: Invalid deserialized string length 1868955648. Maximum size exceeded.[PN5boost16errinfo_at_line_E] = 507
 
Mar 10 11:31:18 axinsadec3 nw[8057]: [stats] [info] Found 7 files (399.61 MB) when loading /var/netwitness/logdecoder/statdb of max size 1 GB
 
Mar 10 11:31:18 axinsadec3 nw[8057]: [ObjectStoreIndex] [warning] 
 Invalid index /var/netwitness/logdecoder/statdb/stats-000000028.statsdbindex. Last object position 16740126 exceeds store size 16740126. Regenerating index... 

Cause

In this example, the Log Decoder crashes were due to a statdb file corruption. It keeps looping over and over whilst trying to start the nwlogdecoder service.

Workaround

As a workaround, delete the core files.
If the core file creation is due to a corrupt statdb file, then move or rename the statdb file mentioned in the error message, and then restart the nwlogdecoder service.

start nwlogdecoder

Backup at least one recent core file in case further an investigation of the issue is needed with RSA Engineering.

Resolution

Ensure the latest NetWitness patch release has been installed.

If NetWitness is running with the latest version, then collect a sample core file from when the issue started to occur, and contact RSA Support to open a case with RSA Engineering to investigate the cause, then proceed to the workaround.

Product Details

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: NetWitness Logs & Network UI
RSA Version/Condition: 10.6.x
Platform: CentOS
O/S Version: 6
Product Name: SA-HYBRID-L
Product Description: SecAnlytcs Hybrd Dplymnt Logs

Summary

The NetWitness Log Decoder service won't stay started, keeps looping to start and creating core dump files.

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue