Skip to content
  • There are no suggestions because the search field is empty.

How to Extract RDQ files generated in NetWitness Log Collector/ Virtual Log Collector in Readable Format

Tasks

Make sure to have nwlogcollectortools rpm installed in Log Collector. You can verify the same using:-
# rpm -qa | grep -i nwlog
If it not installed already then please run below command:-
#yum install nwlogcollectortools

Resolution

Prerequisites:
On certain versions, the required package "rsa-nw-logcollector-tools" is not present by default, so you must check if the packager is already installed and install if not.

To check if it's already installed, run the following command and check for an output similar to this:

[root@NW12-LOG-HYBRID ~]# rpm -qa | grep -i logcollector-tools
rsa-nw-logcollector-tools-12.5.1.0-15145.5.6bc32add1.el8.x86_64

If no RPM package is returned from your query, ou must first install the tools that provide the NwEventReader utility, which are located in our local repo and can be installed with this example: 

yum install rsa-nw-logcollector-tools -y

After a successful install, the following utilities will be available on the command line:

[root@NW12-LOG-HYBRID /]# ls -lrth /usr/bin/ | grep "NwAM\|Event"
-rwxr-xr-x. 1 root root 13M Oct 17 14:52 NwAMQPSender
-rwxr-xr-x. 1 root root 13M Oct 17 14:52 NwEventReader
-rwxr-xr-x. 1 root root 13M Oct 17 14:52 NwAMQPReceiver

NwAMQPReceiver

The NwAMQPReceiver utility will convert RDQ files persisted on disk into NGCE files, which can then be processed by the NwEventReader utility, using these steps:

  1. Create the following directories:
    [root@NW12-LOG-HYBRID /]# mkdir -p /var/netwitness/tmp/ngcefiles
    [root@NW12-LOG-HYBRID /]# mkdir -p /var/netwitness/tmp/rdqfiles
    [root@NW12-LOG-HYBRID /]# mkdir -p /var/netwitness/tmp/rdqlogging

  2.  Copy the RDQ files that have been persisted on-disk from /var/netwitness/rabbitmq/mnesia/rabbit@10452418-1d88-41e7-8995-ab9e522badf0/msg_stores/vhosts/7KP3WGQ2ZYSGFRALFLOZ3NZU/msg_store_persistent/*.rdq to /var/netwitness/tmp/rdqfiles
    1. Note 10452418-1d88-41e7-8995-ab9e522badf0 & 7KP3WGQ2ZYSGFRALFLOZ3NZU are unique IDs and will be different on every host. 
    2. Example:
      [root@NW12-LOG-HYBRID /]# cp /var/netwitness/rabbitmq/mnesia/rabbit@10452418-1d88-41e7-8995-ab9e522badf0/msg_stores/vhosts/7KP3WGQ2ZYSGFRALFLOZ3NZU/msg_store_persistent/*.rdq /var/netwitness/tmp/rdqfiles
      [root@NW12-LOG-HYBRID /]# ls -lrth /var/netwitness/tmp/rdqfiles
      total 33M
      -rw-r-----. 1 root root 0 Mar 7 15:08 0.rdq
      -rwx------. 1 root root 17M Mar 7 15:08 408854.rdq
      -rwx------. 1 root root 17M Mar 7 15:08 408888.rdq
  3. Run the NwAMQPReceiver utility to convert them into NGCE files and confirm they were created. Example:
    [root@NW12-LOG-HYBRID /]# NwAMQPReceiver --fromdir /var/netwitness/tmp/rdqfiles --dir /var/netwitness/tmp/ngcefiles 1> /var/netwitness/tmp/rdqlogging/NwAMQPReceiver.logs
    [root@NW12-LOG-HYBRID /]# ls -lrth /var/netwitness/tmp/ngcefiles | tail -n2
    -rw-r--r--. 1 root root 105K Mar 7 15:11 408888_rdq-174136028564200000133.ngce
    -rw-r--r--. 1 root root 125K Mar 7 15:11 408888_rdq-174136028564200000132.ngce

NwEventReader

The NwEventReader tool can be used to read events stored in protobuf format, either either as streamed to the disk from the Log Collector (using the NGCP protocol), as stored on disk in persistent format as a result of stopping collection, or as pulled from the Message Broker using the NwAMQPReceiver tool.

Syntax:

[root@NW12-LOG-HYBRID ~]# NwEventReader --help
Syntax:
  --help                 print this help message
  --file arg             File to dump.  This may be a file captured from 
                         streamed (NGCP) output from the Log Collector, or 
                         messages captured via the AMQPReceiver tool.
  --printEvents arg (=1) Print events
  --verbose arg (=0)     Verbose output
 
  1. Run the NwEventReader utility to convert the NGCE files to human readable format. Example:
    [root@NW12-LOG-HYBRID /]# NwEventReader --file /var/netwitness/tmp/ngcefiles/408888_rdq-174136028564200000132.ngce --printEvents=1
  2. Example output:
    ======================================================
    Event: 469:
    Event:
    collection_meta:
    "lc.lpid" : "syslog.syslog-tcp"
    "lc.cid" : "LHPRSAVLCV02"
    "lc.msgtype" : "0"
    "lc.ctype" : "syslog"
    "lc.wuid" : "17562157925649023279"
    "lc.wusn" : "255443469"
    "lc.esname" : "tcp514"
    "lc.estype" : "syslog-tcp"
    "lc.ctime" : "1612363394573"
    "lc.srcid" : "10.1.231.88"

    content_meta:
    "syslog.pri" : "4"
    "syslog.level" : "4"
    "syslog.capture.ctime" : "1612363394573"
    "syslog.capture.srcid" : "10.1.231.88"
    "syslog.body" : "%MSWIN-Security-4634: Agent=NWE AgentIP=10.1.231.88 AgentComputer=LHPISMAPPV01 AgentTime=2021-02-03T14:43:14.5009237Z TimeCreatedSystemTime=2021-02-03T14:42:51.7370490Z EventID=4634 Provider="Microsoft Windows security auditing." Channel=Security Level=Information Task=Logoff OpCode=Info Version=0 Keyword="Audit Success" ProcessID=824 Computer=LHPISMAPPV01.dohms.gov.ae RecordId=6371877 TargetUser=DOHMS\sysmonadmin TargetUserName=sysmonadmin TargetDomainName=DOHMS TargetLogonId=0x5419c213 LogonType=3 Message="An account was logged off. Subject: Security ID: S-1-5-21-406404882-2001345335-654838779-49090 Account Name: sysmonadmin Account Domain: DOHMS Logon ID: 0x5419C213 Logon Type: 3 This event is generated when a logon session is destroyed. It may b^C
  3.  To write to a text file, do the following:
    [root@NW12-LOG-HYBRID /]# mkdir -p /var/netwitness/tmp/rawlogs/
    [root@NW12-LOG-HYBRID /]# NwEventReader --file /var/netwitness/tmp/ngcefiles/408888_rdq-174136028564200000132.ngce --printEvents=1 > /var/netwitness/tmp/rawlogs/rawlogs.txt
  4. Confirm the file was generated:
    [root@NW12-LOG-HYBRID /]# tail -n15 /var/netwitness/tmp/rawlogs/rawlogs.txt
    "syslog.pri" : "4"
    "syslog.level" : "4"
    "syslog.capture.ctime" : "1612363395533"
    "syslog.capture.srcid" : "10.1.248.22"
    "syslog.body" : "%MSWIN-Security-4662: Agent=NWE AgentIP=10.1.248.22 AgentComputer=HQPDCP02 AgentTime=2021-02-03T14:43:08.8981363Z TimeCreatedSystemTime=2021-02-03T14:42:43.5119871Z EventID=4662 Provider="Microsoft Windows security auditing." Channel=Security Level=Information Task="Directory Service Access" OpCode=Info Version=0 Keyword="Audit Failure" ProcessID=840 Computer=HQPDCP02.dohms.gov.ae RecordId=2593194261 SubjectUser=DOHMS\MIJawarawala SubjectUserName=MIJawarawala SubjectDomainName=DOHMS SubjectLogonId=0x4f6b104d ObjectServer=DS ObjectType=%{bf967aba-0de6-11d0-a285-00aa003049e2} ObjectName=%{8425ae20-fb01-4b40-90f5-9bce8ec3113a} OperationType="Object Access" HandleId=0x0 AccessList="%%7688 " AccessMask=0x100 Properties="--- {91e647de-d96f-4b70-9557-d63ff4f3ccd8} {6617e4ac-a2f1-43ab-b60c-11fbd1facf05} {b3f93023-9239-4f7c-b99c-6745d87adbc2} {b8dfa744-31dc-4ef1-ac7c-84baf7ef9da7} {b7ff5a38-0818-42b0-8110-d3d154c97f24} {771727b1-31b8-4cdf-ae62-4fe39fadf89e} {612cb747-c0e8-4f92-9221-fdd5f15b550d} {bf967aba-0de6-11d0-a285-00aa003049e2} " AdditionalInfo=- AdditionalInfo2=- Message="An operation was performed on an object. Subject : Security ID: S-1-5-21-406404882-2001345335-654838779-38141 Account Name: MIJawarawala Account Domain: DOHMS Logon ID: 0x4F6B104D Object: Object Server: DS Object Type: %{bf967aba-0de6-11d0-a285-00aa003049e2} Object Name: %{8425ae20-fb01-4b40-90f5-9bce8ec3113a} Handle ID: 0x0 Operation: Operation Type: Object Access Accesses: Control Access Access Mask: 0x100 Properties: --- {91e647de-d96f-4b70-9557-d63ff4f3ccd8} {6617e4ac-a2f1-43ab-b60c-11fbd1facf05} {b3f93023-9239-4f7c-b99c-6745d87adbc2} {b8dfa744-31dc-4ef1-ac7c-84baf7ef9da7} {b7ff5a38-0818-42b0-8110-d3d154c97f24} {771727b1-31b8-4cdf-ae62-4fe39fadf89e} {612cb747-c0e8-4f92-9221-fdd5f15b550d} {bf967aba-0de6-11d0-a285-00aa003049e2} Additional Information: Parameter 1: - Parameter 2: ""
    "syslog.processing.format" : "NonStandard-Syslog: +Pri"

    raw_message: <4> %MSWIN-Security-4662: Agent=NWE AgentIP=10.1.248.22 AgentComputer=HQPDCP02 AgentTime=2021-02-03T14:43:08.8981363Z TimeCreatedSystemTime=2021-02-03T14:42:43.5119871Z EventID=4662 Provider="Microsoft Windows security auditing." Channel=Security Level=Information Task="Directory Service Access" OpCode=Info Version=0 Keyword="Audit Failure" ProcessID=840 Computer=HQPDCP02.dohms.gov.ae RecordId=2593194261 SubjectUser=DOHMS\MIJawarawala SubjectUserName=MIJawarawala SubjectDomainName=DOHMS SubjectLogonId=0x4f6b104d ObjectServer=DS ObjectType=%{bf967aba-0de6-11d0-a285-00aa003049e2} ObjectName=%{8425ae20-fb01-4b40-90f5-9bce8ec3113a} OperationType="Object Access" HandleId=0x0 AccessList="%%7688 " AccessMask=0x100 Properties="--- {91e647de-d96f-4b70-9557-d63ff4f3ccd8} {6617e4ac-a2f1-43ab-b60c-11fbd1facf05} {b3f93023-9239-4f7c-b99c-6745d87adbc2} {b8dfa744-31dc-4ef1-ac7c-84baf7ef9da7} {b7ff5a38-0818-42b0-8110-d3d154c97f24} {771727b1-31b8-4cdf-ae62-4fe39fadf89e} {612cb747-c0e8-4f92-9221-fdd5f15b550d} {bf967aba-0de6-11d0-a285-00aa003049e2} " AdditionalInfo=- AdditionalInfo2=- Message="An operation was performed on an object. Subject : Security ID: S-1-5-21-406404882-2001345335-654838779-38141 Account Name: MIJawarawala Account Domain: DOHMS Logon ID: 0x4F6B104D Object: Object Server: DS Object Type: %{bf967aba-0de6-11d0-a285-00aa003049e2} Object Name: %{8425ae20-fb01-4b40-90f5-9bce8ec3113a} Handle ID: 0x0 Operation: Operation Type: Object Access Accesses: Control Access Access Mask: 0x100 Properties: --- {91e647de-d96f-4b70-9557-d63ff4f3ccd8} {6617e4ac-a2f1-43ab-b60c-11fbd1facf05} {b3f93023-9239-4f7c-b99c-6745d87adbc2} {b8dfa744-31dc-4ef1-ac7c-84baf7ef9da7} {b7ff5a38-0818-42b0-8110-d3d154c97f24} {771727b1-31b8-4cdf-ae62-4fe39fadf89e} {612cb747-c0e8-4f92-9221-fdd5f15b550d} {bf967aba-0de6-11d0-a285-00aa003049e2} Additional Information: Parameter 1: - Parameter 2: "


    Summary:
    ========
    Read 1000 events in 126991 bytes.
    compression ratio (u/c): 29.6319



 


Notes


Product Details

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Log Decoder, Log Collector
NetWitness Version/Condition: 12.x
Platform: CentOS/AlmaLinux


Summary

You can verify if the correct and required RDQs are reaching to collector or not using NwEventReader Tool.


Approval Reviewer Queue

Technical approval queue