Parsing a tab-delimited log file into the NetWitness Platform

Issue

This article describes how to parse a tab delimited log file into the NetWitness Platform.

The log file is of the following form and is tab delimited: (Note: the contents below should all be on a single line.)

#Fields: datatime c-ip x-ss-company-id cs(X-Forwarded-For) cs-username cs-method cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs(User-Agent)
cs(Content-Type) cs-bytes sc-bytes sc-status sc(Content-Type) s-ip x-ss-category x-ss-last-rule-name x-ss-last-rule-action x-ss-block-type x-ss-block-value
x-ss-external-ip x-ss-referer-host 2015-07-10 11:39:37 GMT 10.106.21.99 2164457336 10.106.21.99 CONNECT https www.ibm.com 443 / curl/7.19.7 (x86_64-redhat-linux-gnu)
libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - 0 0 0 23.32.171.219 c:busi default block adv-rule-match No exception exists to allow
this web page 128.221.224.200 2015-07-10 11:41:40 GMT 10.106.21.99 2164457336 10.106.21.99 CONNECT https www.ibm.com 443 / curl/7.19.7 (x86_64-redhat-linux-gnu)
libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - 0 0 0 23.32.171.219 c:busi default block adv-rule-match No exception exists to allow
this web page 168.159.213.199

Tasks

This procedure uploads the file to a Log Collector using the File collection method. This enables replacing the TAB delimiters with an alternative delimiter and also append some text onto the start of each line of the log to assist with parsing.

Create a New Event Source File Type

On the Log Collector create the following file /etc/netwitness/ng/logcollection/content/collection/file/ciscocwsctm.xml

 <?xml version="1.0" encoding="UTF-8"?>
 
<typespec>
 
 <name>ciscocwsctm</name>
 
 <type>file</type>
 
 <prettyName>ciscocwsctm</prettyName>
 
 <version>1.0</version>
 
 <author>administrator</author>
 
 <description>FileCollection specification for eventsource type Cisco CWS Custom using file handler type "ciscocwsctm"</description>
 
 <device>
 
 <name>ciscocwsctm</name>
 
 </device>
 
 <configuration>
 
 </configuration>
 
 <collection>
 
 <file>
 
 <parserId>file.ciscocwsctm</parserId>
 
 <processorType>generic</processorType>
 
 <dataStartLine>2</dataStartLine>
 
 <fieldDelim>0x09</fieldDelim>
 
 <idField></idField>
 
 <lineDelim>\n</lineDelim>
 
 <transformPrefixTag>CISCOCWSCTM</transformPrefixTag>
 
 <transformReplaceFieldDelim>1</transformReplaceFieldDelim>
 
 <transformPrefixFilename>0</transformPrefixFilename>
 
 <transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
 
 <transformReplacementFieldDelim>^^</transformReplacementFieldDelim>
 
 </file>
 
 </collection>
 
</typespec>

Important points about this file:

Replace TAB delimiters (0x09) with ^^ delimters to aid parsing
Add %CISCOWSCTM to each line of the log file to aid parsing
Restart the logcollector service (restart nwlogcollector) to apply this or the changes to take effect.

Create a New Parser

Create a new directory called /etc/netwitness/ng/envision/etc/devices/ciscocwsctm
on the logdecoder.

Copy the attached files to this location

/etc/netwitness/ng/envision/etc/devices/ciscocwsctm/ciscocwsctm.ini
/etc/netwitness/ng/envision/etc/devices/ciscocwsctm/ciscocwsctmmsg.xml

Edit /etc/netwitness/ng/envision/etc/table-map-custom.xml xml configuration file to include the following:

 <!--My Custom Parser --> 
 
<mapping envisionName="csusername" nwName="user.src" flags="None"/> 
 
<mapping envisionName="xforwardfor" nwName="xfactor" flags="None"/> 
 
<mapping envisionName="csmethod" nwName="web_method" flags="None"/> 
 
<mapping envisionName="csurischeme" nwName="protocol" flags="None"/> 
 
<mapping envisionName="cshost" nwName="alias.host" flags="None"/> 
 
<mapping envisionName="csuriport" nwName="dport" flags="None"/> 
 
<mapping envisionName="csuripath" nwName="url.raw" flags="None"/> 
 
<mapping envisionName="csuriquery" nwName="query" flags="None"/> 
 
<mapping envisionName="csuseragent" nwName="user.agent" flags="None"/> 
 
<mapping envisionName="cscontenttype" nwName="content" flags="None"/> 
 
<mapping envisionName="csbytes" nwName="sbytes" flags="None"/> 
 
<mapping envisionName="scbytes" nwName="rbytes" flags="None"/> 
 
<mapping envisionName="scstatus" nwName="result.code" flags="None"/> 
 
<mapping envisionName="sccontenttype" nwName="content" flags="None"/> 
 
<mapping envisionName="xsscategory" nwName="category" flags="None"/> 
 
<mapping envisionName="xssname" nwName="policy.name" flags="None"/> 
 
<mapping envisionName="xsslastruleaction" nwName="action" flags="None"/> 
 
<mapping envisionName="xssblocktype" nwName="policy.name" flags="None"/> 
 
<mapping envisionName="xssblockvalue" nwName="result" flags="None"/> 
 
<mapping envisionName="xssexternalip" nwName="dtransaddr" flags="None"/> 
 
<mapping envisionName="xssreferhost" nwName="referer" flags="None"/> 
 
<!-- My Custom Parser -->

Restart the log decoder with

 restart nwlogdecoder

Resolution

The parsing works because the log file is in a simple structured format.
This means that the parser only needs to expect one message.

This is defined as follows in the ciscowsctmmsg.xml file: (Note: the contents below should all be on a single line.)

content="<saddr>^^<xsscompanyid>^^<xforwardfor>^^<csusername>^^<csmethod>^^<csurischeme>^^<cshost>^^<csuriport>^^<csuripath>
^^<csuriquery>^^<csuseragent>^^<cscontenttype>^^<csbytes>^^<scbytes>^^<scstatus>^^<scontenttype>^^<daddr>^^<xsscategory>^^<
xssname>^^<xsslastruleaction>^^<xssblocktype>^^<xssblockvalue>^^<xssexternalip>^^<xssreferhost>"/>

The table-map-custom.xml file maps these fields into the NetWitness meta keys.

Product Details

RSA Product Set: Security Analytics, NetWitness
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.x
Platform: CentOS
O/S Version: 6

Summary

How to parse a tab delimited log file into Security Analytics.

Approval Reviewer Queue

Technical approval queue