Logdecoder Capture stops intermittently due to less metadb size in RSA Security Analytics
Issue
Logdecoder capture stops intermittently with below errors.
/var/log/messages:
Sep 10 06:40:17 TCLDecDot NwLogDecoder[32074]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.65 GB free), capture is stopping. Please check drive and configuration.
Sep 19 08:52:29 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.92 GB free), capture is stopping. Please check drive and configuration.
Sep 20 03:27:44 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.99 GB free), capture is stopping. Please check drive and configuration.
Sep 20 13:26:35 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.69 GB free), capture is stopping. Please check drive and configuration.
Sep 10 06:40:17 TCLDecDot NwLogDecoder[32074]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.65 GB free), capture is stopping. Please check drive and configuration.
Sep 19 08:52:29 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.92 GB free), capture is stopping. Please check drive and configuration.
Sep 20 03:27:44 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.99 GB free), capture is stopping. Please check drive and configuration.
Sep 20 13:26:35 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.69 GB free), capture is stopping. Please check drive and configuration.
In this instance, the meta.free.space.min value in Logdecoder->Explore->Database->Config page is 3 GB
Cause
The usage of metadb/sessiondb/packetdb/indexdb which even if grows beyond the configured size is a normal scenario as long as rollover is occurring automatically before the filesystem fills, it is functioning as designed.As rollover is not that precise and rollover is only active once the usage exceeds the specified size threshold, and only is activated periodically, rather than instantaneously.
So,it seems rollover starts periodically and in that mean time db grows more than 95%. This causes the free space available for core database directories getting reduced. But, the core services work when minimum required free space available.
Resolution
In this circumstance, The metadb should have minimum 3 GB free space to work logdecoder service. The log errors show capture stopped details when meta free space reduced to <3 GB. So, the log pattern says 2 GB meta free space setting is a good idea, Since the free space never reduced <2 GB.Please follow below steps to solve this issue permanently.
1. Login to GUI and Navigate to Logdecoder->Explore view.
2. Left hand side expand database->config
3. Chang meta.free.space.min value from 3 GB to 2 GB.
This change would take effect immediately.
Product Details
RSA Product Set: NetWitnessRSA Product/Service Type: NetWitness
RSA Version/Condition: 11.x 12.x
Summary
In Logdecoder, the capture stops intermittently when less space available for metadb. This can be fixed by adjusting meta.free.space.min value.
Approval Reviewer Queue
Technical approval queue