.png?width=180&height=32&name=NW%20Logo%20(2).png){kind=small}
NetWitness Data Retention report
Issue
How to produce a report that shows each of the NetWitness database settings and how much data each retains compared to free disk space?Want to see current retention days, check that the configuration is correct for the available disk space.
This is the Archiver, Broker, Concentrator, Decoder, Log Decoder databases.
Where the databases for each service are the following.
Archiver, Decoder, Log Decoder: index, meta, packet, session
Broker: index
Concentrator: index, meta, session
Resolution
Copy the attached retention.sh script to the Archiver, Broker, Concentrator, Decoder, or Log Decoder appliance under /root directory.Make the script executable.
chmod +x ./retention.sh
Run the script to produce the report.
./retention.sh
For example:
[root@NWCONC ~]# ./retention.sh
*** NWCONC RETENTION REPORT - Version:4 Tue Nov 9 06:09:11 UTC 2021 ***
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/netwitness_vg00-nwhome 2802847 4754 2798094 1% /var/netwitness
Concentrator
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/concentrator-root 30705 3480 27226 12% /var/netwitness/concentrator
/dev/mapper/index-index 950544 74043 876502 8% /var/netwitness/concentrator/index
/dev/mapper/concentrator-metadb 24000842 986192 23014651 5% /var/netwitness/concentrator/metadb
/dev/mapper/concentrator-sessiondb 2669448 69180 2600269 3% /var/netwitness/concentrator/sessiondb
index:
time.begin: 2010-Feb-09
Configured: /var/netwitness/concentrator/index=835.44GB
Used=74006M /var/netwitness/concentrator/index
Largest=11243 MB (managed-values-26), free disk space=876502 MB on mount /var/netwitness/concentrator/index
Oldest file/dir: Jul 12 2019 /var/netwitness/concentrator/index/managed-values-0
/var/log/messages:
Sep 16 20:59:38 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/index.
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounting /var/netwitness/concentrator/index...
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/index.
meta:
meta.oldest.file.time: 2019-Jul-01
Configured: /var/netwitness/concentrator/metadb=21.74TB
meta.free.space.min: 203GB (207872MB)
Used=986159M /var/netwitness/concentrator/metadb
Largest=3073 MB (meta-000000283.nwmdb), free disk space=23014651 MB on mount /var/netwitness/concentrator/metadb
Oldest file/dir: Jul 5 2019 /var/netwitness/concentrator/metadb/meta-000000001.nwmdbindex
/var/log/messages:
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounting /var/netwitness/concentrator/metadb...
Oct 30 01:34:33 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/metadb.
Oct 30 01:34:37 CS-NWCON-31 NwConcentrator[1292]: [meta] [info] Found 320 files (904.99 GB) when loading /var/netwitness/concentrator/metadb of max size 21.74 TB
session:
session.oldest.file.time: 2019-Jul-01
Configured: /var/netwitness/concentrator/sessiondb=2.42TB
session.free.space.min: 22GB (22528MB)
Used=69148M /var/netwitness/concentrator/sessiondb
Largest=1537 MB (session-000000051.nwsdb), free disk space=2600269 MB on mount /var/netwitness/concentrator/sessiondb
Oldest file/dir: Oct 30 01:34 /var/netwitness/concentrator/sessiondb/session-000000001.nwsdb
/var/log/messages:
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounting /var/netwitness/concentrator/sessiondb...
Oct 30 01:34:34 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/sessiondb.
Oct 30 01:34:37 CS-NWCON-31 NwConcentrator[1292]: [session] [info] Found 64 files (67.52 GB) when loading /var/netwitness/concentrator/sessiondb of max size 2.42 TB
High Retention Days = 862 and Meta/Session days = 862
*** NWCONC RETENTION REPORT - Version:4 Tue Nov 9 06:09:11 UTC 2021 ***
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/netwitness_vg00-nwhome 2802847 4754 2798094 1% /var/netwitness
Concentrator
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/concentrator-root 30705 3480 27226 12% /var/netwitness/concentrator
/dev/mapper/index-index 950544 74043 876502 8% /var/netwitness/concentrator/index
/dev/mapper/concentrator-metadb 24000842 986192 23014651 5% /var/netwitness/concentrator/metadb
/dev/mapper/concentrator-sessiondb 2669448 69180 2600269 3% /var/netwitness/concentrator/sessiondb
index:
time.begin: 2010-Feb-09
Configured: /var/netwitness/concentrator/index=835.44GB
Used=74006M /var/netwitness/concentrator/index
Largest=11243 MB (managed-values-26), free disk space=876502 MB on mount /var/netwitness/concentrator/index
Oldest file/dir: Jul 12 2019 /var/netwitness/concentrator/index/managed-values-0
/var/log/messages:
Sep 16 20:59:38 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/index.
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounting /var/netwitness/concentrator/index...
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/index.
meta:
meta.oldest.file.time: 2019-Jul-01
Configured: /var/netwitness/concentrator/metadb=21.74TB
meta.free.space.min: 203GB (207872MB)
Used=986159M /var/netwitness/concentrator/metadb
Largest=3073 MB (meta-000000283.nwmdb), free disk space=23014651 MB on mount /var/netwitness/concentrator/metadb
Oldest file/dir: Jul 5 2019 /var/netwitness/concentrator/metadb/meta-000000001.nwmdbindex
/var/log/messages:
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounting /var/netwitness/concentrator/metadb...
Oct 30 01:34:33 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/metadb.
Oct 30 01:34:37 CS-NWCON-31 NwConcentrator[1292]: [meta] [info] Found 320 files (904.99 GB) when loading /var/netwitness/concentrator/metadb of max size 21.74 TB
session:
session.oldest.file.time: 2019-Jul-01
Configured: /var/netwitness/concentrator/sessiondb=2.42TB
session.free.space.min: 22GB (22528MB)
Used=69148M /var/netwitness/concentrator/sessiondb
Largest=1537 MB (session-000000051.nwsdb), free disk space=2600269 MB on mount /var/netwitness/concentrator/sessiondb
Oldest file/dir: Oct 30 01:34 /var/netwitness/concentrator/sessiondb/session-000000001.nwsdb
/var/log/messages:
Oct 30 01:34:32 CS-NWCON-31 systemd: Mounting /var/netwitness/concentrator/sessiondb...
Oct 30 01:34:34 CS-NWCON-31 systemd: Mounted /var/netwitness/concentrator/sessiondb.
Oct 30 01:34:37 CS-NWCON-31 NwConcentrator[1292]: [session] [info] Found 64 files (67.52 GB) when loading /var/netwitness/concentrator/sessiondb of max size 2.42 TB
High Retention Days = 862 and Meta/Session days = 862
Notes about the script output.
1. Displays the disk usage of the /var/netwitness mount, in case this is the only mount for all database data (usually VMs).
2. For each service (archiver, broker, concentrator, decoder) display disk usage of any mounts matching that service.
3. For each database (index, meta, packet, session) display the following applicable information.
a. Oldest data date (time.begin, *.oldest.file.time)
b. Configured service mounts and the amount of disk space can use in each mount.
c. Configured minimum free space (*.free.space.min).
When the free disk space falls below this threshold the service will stop aggregating/consuming.
d. Current disk space used under the database's directory.
e. The largest file/directory on the mount, displayed in green if there is sufficient free disk space.
Displayed in red if the current free disk space is less than (2x largest file/directory size + minimum free disk space).
f. The oldest file under the service's mount.
Usually, similar date to oldest data date unless the oldest file was modified, like when using the NwConsole dbcheck.
g. Display a few /var/log/messages log entries for any database mount reference, or any deleting of database files, which indicates database rollout has occurred.
b. Configured service mounts and the amount of disk space can use in each mount.
c. Configured minimum free space (*.free.space.min).
When the free disk space falls below this threshold the service will stop aggregating/consuming.
d. Current disk space used under the database's directory.
e. The largest file/directory on the mount, displayed in green if there is sufficient free disk space.
Displayed in red if the current free disk space is less than (2x largest file/directory size + minimum free disk space).
f. The oldest file under the service's mount.
Usually, similar date to oldest data date unless the oldest file was modified, like when using the NwConsole dbcheck.
g. Display a few /var/log/messages log entries for any database mount reference, or any deleting of database files, which indicates database rollout has occurred.
4. The report ends with two numbers for retention days.
a. The number of days between the current date and the most recent date of the Oldest data date (time.begin, *.oldest.file.time) for index, meta, packet, session.
Where choosing retention days less than 10 as low (red), between 10 and less then 60 as moderate (yellow), and 60 days and above as high (green) retention.
b. The number of days between the current date and the most recent date of the Oldest data date (*.oldest.file.time) for meta, session.
This is the oldest number of days that the Archiver/Concentrator can consume from a Decoder/Log Decoder.
Where choosing retention days less than 10 as low (red), between 10 and less then 60 as moderate (yellow), and 60 days and above as high (green) retention.
b. The number of days between the current date and the most recent date of the Oldest data date (*.oldest.file.time) for meta, session.
This is the oldest number of days that the Archiver/Concentrator can consume from a Decoder/Log Decoder.
Product Details
RSA Product Set: RSA NetWitness PlatformRSA Product/Service Type: Archiver, Broker, Concentrator, Decoder, Log Decoder
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7
Summary
How to show NetWitness data retention details?
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue