LUA - Common Event Format (CEF) Script Template

The CEF helper script attempts to fill the gap of CEF extensions that the system parser does not currently parse. For a complete list of what we do parse, please refer to Supported CEF Meta Keys - RSA Security Analytics Documentation .

This CEF helper template was written to be highly configurable as well as adhere to revision 16 of the Common Event Format (CEF) standards document. Its aim is to be able to parse meta data into any meta key in RSA Netwitness from any security appliance with the least amount of programming.

This is performed by several routines within the code that associates “csX” and “cnX” within a CEF extension with their appropriate “csXLabel” and “cnXLabel”. If no “csLabel” exists, it will use “csX” or “cnX” as the key name (where X is a numeric value). For example, a CEF message containing “cs3=name cs3Label=Ian” will set the key name to “name” and the value to “Ian” whereas a CEF message containing “cs3=Ian” will set the key name to “cs3” and the value to “Ian”. However, using scripts configuration, you can translate cs3 to any meta key name you want (i.e. username). This is configured in the t_keys_to_use variable.  More information on that below!

As CEF messages can also contain carriage returns (\n), if a carriage return is found, it will parse each line into a separate meta value of the same key name. For example, a CEF message containing “cs2=user.names cs2Label=iredden\nepartington” will result in 2 meta values (user.names) containing “iredden” and “epartington”.

The template can be downloaded at the bottom of this article.  It is configured for a FireEye HX appliance but can be easily modified for anything!

Configuration

The main configuration of the script is in 2 variables.  These variables are t_keys_to_use and b_debug.

t_keys_to_use – Is a LUA key/value table containing which keys to parse.

b_debug – By default, this variable is set to true.  This means that no meta will be created.  Instead, output will be provided for debugging to logs.

You also need to configure the cefhelper:setKeys() section of the script.  It needs to contain all the same keys from the t_keys_to_use table.  For example:

cefhelper:setKeys({
    nwlanguagekey.create("vx.threatscore"),
    nwlanguagekey.create("vx.detection"),
    nwlanguagekey.create("virusname"),
   ...
    nwlanguagekey.create("alias.host"),
    nwlanguagekey.create("vx.filedesc")
})

You can access logs on a Log Decoder via the REST API:

http://de.co.der.ip:50102/logs?msg=pull&force-content-type=text/plain&expiry=600&count=50 

Example - Accellion File Transfer (FTA)

The CEF system parser from RSA Live parses numerous CEF extension keys.  However, there are a few meta keys that we need a helper for to be able to parse CEF logs from Accellion FTA:

Helper Configuration:

local t_keys_to_use = {
        ["fname"] = "filename",
        ["location"] = "directory",

        ["type"] = "action"
}

cefhelper:setKeys({
   nwlanguagekey.create("filename"),
   nwlanguagekey.create("directory"),
   nwlanguagekey.create("action")
})

local b_debug = false

Example CEF Message:

CEF:0|Accellion|FTA|FTA_9_12_80|203|Download|1|msg=(172.16.20.45) test.txt downloaded by ian.redden@rsa.local (23 bytes, 3.5 sec) (Type: ssl_download, Location: ) src=172.16.20.45 deviceFacility=local5 fname=test.txt fsize=23 cs2Label=location cs2= cs1Label=type cs1=ssl_download cn1Label=seconds cn1=3.5 suser=ian.redden@rsa.local

Example - Payload Security's VxStream

The CEF system parser from RSA Live parses numerous CEF extension keys.  However, there are a few meta keys that we need a helper for to be able to parse CEF logs from Accellion FTA:

Helper Configuration:

local t_keys_to_use = {
   ["Threat Score"] = "vx.threatscore",
   ["AV Detection Rate"] = "vx.detection",
   ["Malware Family"] = "virusname",
   ["fileHash"] = "vx.filehash",
   ["fname"] = "vx.fname",
   ["fsize"] = "vx.fsize",
   ["fileType"] = "vx.filetype",
   ["flexString1"] = "vx.comments",
   ["Client"] = "vx.client",
   ["Indicators"] = "vx.indicators",
   ["Contacted Domains"] = "alias.host",
   ["Contacted Hosts"] = "alias.host",
   ["Compromised Hosts"] = "alias.host",
   ["File Description"] = "vx.filedesc"
}

cefhelper:setKeys({
   nwlanguagekey.create("vx.threatscore"),
   nwlanguagekey.create("vx.detection"),
   nwlanguagekey.create("virusname"),
   nwlanguagekey.create("vx.filehash"),
   nwlanguagekey.create("vx.fname"),
   nwlanguagekey.create("vx.fsize"),
   nwlanguagekey.create("vx.filetype"),
   nwlanguagekey.create("vx.comments"),
   nwlanguagekey.create("vx.client"),
   nwlanguagekey.create("vx.indicators"),
   nwlanguagekey.create("alias.host"),
   nwlanguagekey.create("vx.filedesc")
})

local b_debug = false

Example CEF Message:

Mar 29 15:42:59 192.168.1.100 CEF:0|Payload Security|VxStream|6.20|Sample Analysis Result - Malicious|Sample Analysis Result - Malicious|100|end=03/29/2017 21:28:55 cn1=100 cn1Label=Threat Score cn2=88 cn2Label=AV Detection Rate cs1=Trojan.Generic cs1Label=Malware Family cs2=100 cs2Label=EnvironmentID cs3=Windows 7 32 bit cs3Label=Environment Description fileHash=8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572 fname=8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572 fsize=357888 fileType=exe request=https://www.hybrid-analysis.com/sample/8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572/?environmentId\=100 msg=Malicious flexString1= flexString1Label=Uploader Comment cs4=zpr5huq4bgmutfnf.tor2web.org \nzpr5huq4bgmutfnf.onion.to \ncrl2.alphassl.com \nipinfo.io \nkosdfnure75.op1gifsd05mllk.com \ngfdkotriam.fo4j4wnq51hepa.com cs4Label=Contacted Domains cs5=84.200.69.80:53 \n34.196.176.140:80 \n185.100.85.150:443 \n192.36.27.5:443 cs5Label=Contacted Hosts cs6=84.200.69.80 \n185.100.85.150 \n192.36.27.5 cs6Label=Compromised Hosts cs8=2812134 \n2015576 \n2812134 \n2020116 \n2020716 \n2016810 \n2016810 cs8Label=ET Alerts cs9=api-12:1:4 \napi-75:1:10 \nsuricata-2:2:10 \nnetwork-21:2:10 \napi-10:1:6 \ntarget-4:2:10 \nnetwork-0:0:1 \nnetwork-2:0:5 \ntarget-25:0:3 \nnetwork-22:1:10 \nregistry-25:1:3 \nstatic-3:1:10 \ntarget-58:2:10 \nstatic-6:1:10 \navtest-3:2:10 \nsuricata-1:1:10 \napi-25:1:7 \nnetwork-27:2:10 \nregistry-27:1:10 \nnetwork-14:2:9 \napi-16:0:1 \nregistry-36:1:8 \nstring-14:1:3 \napi-51:1:5 \nr