Skip to content
  • There are no suggestions because the search field is empty.

NetWitness Database ID search and gap Report

Issue

Want to identify which NetWitness database file contains a particular ID.

Want to produce a report showing any ID gaps in the NetWitness *.manifest files which might identify missing NetWitness database files.

This is for the Archiver, Broker, Concentrator, Decoder, Log Decoder databases.

Where the databases for each service are the following.

Archiver, Decoder, Log Decoder: index, meta, packet, session
Broker: index
Concentrator: index, meta, session

Resolution

Copy the attached db.sh script to the Archiver, Broker, Concentrator, Decoder, or Log Decoder appliance under the /root directory.

Make the script executable.

chmod +x ./db.sh

1. Run the script to show the script syntax.

./db.sh

For example:
[root@NWPKTD ~]# ./db.sh
*** NWPKTD DB REPORT - Version:3 Wed Nov 10 23:57:21 UTC 2021 ***

Script: ./db.sh
Syntax:
-c : Create the temporary files listing the available *.manifest files content
Must run the create option at least once before using the other switches
-f [number] : Find which files contain the provided ID number
-g : Check for ID gaps in the manifest files

2. The script must be run the first time with the create (-c) switch to collect details from all the available *.manifest files on the appliance.

Once the create has completed then the script can be used multiple times with the other switch options.

Run the script with the create (-c) switch again if the manifest information has changed on the appliance.

For example:
[root@NWPKTD ~]# ./db.sh -c
*** NWPKTD DB REPORT - Version:3 Thu Nov 11 00:02:44 UTC 2021 ***

Create Manifest List:
Creating /tmp/manifest-decoder-index.list
Creating /tmp/manifest-decoder-meta.list
Creating /tmp/manifest-decoder-packet.list
Creating /tmp/manifest-decoder-session.list

3. Use the find (-f [number}) switch to find details about which NetWitness database file an ID appear in.

If the ID is found, then the manifest details about the file is shown and a listing of the file if it exists on the appliance.

If the ID is outside of the available manifest files range of IDs, then details about the first or last range of IDs is shown.

For example:
[root@NWPKTD ~]# ./db.sh -f 1234567890
*** NWPKTD DB REPORT - Version:3 Thu Nov 11 00:10:03 UTC 2021 ***

Find ID: 1234567890




ID not found in any decoder index manifest file
ID 1234567890 occurred after the newest managed-values-606.manifest file Range:847058761-847058761

Found in meta
meta-000000007.nwmdb Range:1115963860-1338264581 Wed Jul 10 21:46:13 UTC 2019-Fri Jul 12 07:14:05 UTC 2019
-rw-------. 1 root root 3.1G Feb 11 2020 /var/netwitness/decoder/metadb/meta-000000007.nwmdb

ID not found in any decoder packet manifest file
ID 1234567890 occurred before the oldest packet-000035113.pcapng.manifest file Range:495938412447-495951570753

ID not found in any decoder session manifest file
ID 1234567890 occurred after the newest session-000000082.nwsdb.manifest file Range:847043576-847058761

4. Use the gap (-g) switch to report any gaps in ID ranges in the .manifest files.

Any ID gap found will display details about the NetWitness database files before and after the gap.

For example:
[root@NWPKTD ~]# ./db.sh -g
*** NWPKTD DB REPORT - Version:3 Thu Nov 11 00:20:38 UTC 2021 ***

Check Gaps:




decoder index:
607 manifest files listed in file /tmp/manifest-decoder-index.list

decoder meta:
140 manifest files listed in file /tmp/manifest-decoder-meta.list

decoder packet:
4176 manifest files listed in file /tmp/manifest-decoder-packet.list

decoder session:
Gap: decoder filename:session-000000078.nwsdb id1:846970089 id2: 846970921 time1:Thu Sep 30 15:50:38 UTC 2021
Gap: decoder filename:session-000000079.nwsdb id1: 846970929 id2:846971148 time1:Thu Oct 21 22:28:36 UTC 2021
---
82 manifest files listed in file /tmp/manifest-decoder-session.list

Warning: This script will be slow to run on appliances which have a large number of .manifest files.

Internal Comments

I am unable to track down the old db.sh script anywhere. I've looked in JIRA, SF and reached out across NW Answers and CS, but no one still has the original copy, so unfortunately I'm going to have to Archive this. If the script can be found anywhere, we can update and re-post.


Product Details

RSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: Archiver, Broker, Concentrator, Decoder, Log Decoder
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7

Summary

How to find which NetWitness database file contains an ID and how to check if there are any gaps?


Approval Reviewer Queue

Technical approval queue