Configure Logstash Event Sources in NetWitness

You can configure the Logstash collection protocol.

IMPORTANT:
- Do not change logstash.yml file as it breaks the functionality.
- Do not change sincedb_path input configuration. If you change the sincedb_path, the back up and restore functionality breaks.
- Do not modify any pipeline configuration yml files.

To configure a Logstash Event Source:

Go to (Admin) > Services from the NetWitness menu.
Select a Log Collection service.
Under Actions, select > View > Config to display the Log Collection configuration parameter tabs.
Click the Event Sources tab.

In the Event Sources tab, select Logstash/Config from the drop-down menu.
In the Event Categories panel toolbar, click .

The Available Event Source Types dialog is displayed.
Select the event source type and click OK.

The newly added event source type is displayed in the Event Categories panel.
Select the new type in the Event Categories panel and click in the Sources toolbar.

The Add Source dialog is displayed.
Fill in the fields, based on the Logstash event source you are adding. General details about the available parameters are described below in Logstash Collection Parameters.
Click OK.

Logstash Collection Parameters

The following tables provides descriptions of the Logstash Collection source parameters.

Note: Items that are followed by an asterisk (*) are required.

Basic Parameters

Custom Event Source Parameters

The following table lists the custom event source parameters.

Name:
Name *
Description:
Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description:
Select the checkbox to enable the event source configuration to start collection. The checkbox is selected by default.

Name:
Description
Description:
Enter a text description for the event source.

Name:
Input Configuration *
Description:
An input plugin enables a specific source of events to be read by Logstash.

Name:
Filter Configuration *
Description:
A filter plugin performs intermediary processing on an event. Paste in any filter details for your Logstash event source.

Name: Device Type *
Description:
Enter the device parser type used to parse the data.

Note:
- While saving an existing instance with no specified device type, enter the device type to enable Ok. Else, click Cancel.
- The device type can have 3 to 30 characters of a-z, 1-9, or underscore and must start with a-z.

Name: Source Address
Description: Enter the IP address, host name or other identifier of the event source.

Name: Message ID
Description:
Enter the message group ID used to bypass header parsing.

Name: Message Prefix
Description:
Enter the prefix added to each message to assist parsing.

Name:
Event Destination *
Description:
Select the NetWitness Log Collector or Log Decoder to which event needs to be send from the drop-down list.

Name:
Test Configuration
Description:
Checks the configuration parameters specified in this dialog to make sure they are correct.

Beats Event Source Parameters

The following table lists the beats event source parameters.

Name:
Name *
Description: Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description: Select the check box to enable the event source configuration to start collection. The check box is selected by default.

Name:
Description
Description: Enter a text description for the event source.

Name:
Port Number*
Description: Enter the port number (for example, 5044) that you configured for your event sources.

Name:
Linux-Audit
Description: Select the checkbox to enable processing for Linux audit.

Name: Linux-System
Description: Select the checkbox to enable processing for Linux system.

Name: Ngnix
Description: Select the checkbox to enable processing for Nginx.

Name: Event Destination*
Description: Select the NetWitness Log Collector or Log Decoder to which event needs to be send from the drop-down list.

Name:
Test Configuration
Description:
Checks the configuration parameters specified in this dialog to make sure they are correct.

Export Connector Event Source Parameters

The following table lists the custom export connector event source parameters.

Name:
Name *
Description: Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description: Select the check box to enable the event source configuration to start collection. The check box is selected by default.

Name:
Description
Description: Enter a text description for the event source.

Name:
Host*
Description: Select the hostname of the Decoder or Log Decoder for data aggregation from the drop-down list.

Name:
Username*
Description: Username used to access the Decoder or Log Decoder for data aggregation.

Name: Authentication
Description:
Note: If you upgrade from NetWitness Platform 11.6.0.0 to 11.6.1.0, automatic key is generated and stored in the key store management for the password set in Logstash pipeline configuration. You can view the key instead of password in the Authentication field.

Select the authentication type used for data aggregation. By default, SSL field is enabled, if you select trusted authentication.

Note: For trusted authentication, make sure you add the PEM file at /etc/pki/nw/node/node-cert.pem to the source Decoders REST APIs (/sys/trustpeer and /sys/caupload).

Name:
SSL
Description:
Select the check box to communicate using SSL. The security of data transmission is managed by encrypting information and providing authentication with SSL certificates. By default, SSL option is enabled, if you select trusted authentication type in the Authentication field.

Name:
Decoder Type*
Description:
Decoder Type is a read only field and it is auto populated when you select the Host.

Name: Output Configuration*
Description:
Logstash pipeline output configuration to forward events received from input stream to a defined destination/s. The output plugin sends the processed event data to the data warehouse destinations. You can use the standard Logstash output plugins to send the data. To understand more, see Work Flow of NetWitness Export Connector.

A basic sample Logstash output TCP plugin would look like below.

output {

tcp {

id => "nw-output-tcp"

host => "10.10.1.2"

port => 514

}

}

You can use any of the listed output plugins on Output plugins. We recommend you check with your respective vendor to know the input receiver type (such as TCP, HTTP) they support.

Name:
Test Configuration
Description:
Checks the configuration parameters specified in this dialog to make sure they are correct.

HTTP Receiver Event Source Parameters

The following table lists the HTTP receiver event source parameters.

Name:
Name *
Description: Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description: Select the checkbox to enable the event source configuration to start collection. The checkbox is selected by default.

Name:
Description
Description: Enter a text description for the event source.

Name:
Port Number*
Description: Enter the port number that you configured for your event sources. The default value of port number is 8080.

Name:
Device Type*
Description:
Enter the device parser type used to parse the data.

Name: Message ID
Description:
Enter the message group ID used to bypass header parsing.

Name:
Message Prefix
Description:
Enter the prefix added to each message to assist parsing.

Name:
Event Destination*
Description:
Select the NetWitness Log Collector or Log Decoder to which event needs to be sent from the drop-down list.

Name: HTTP Receiver SSL
Description:
Select the checkbox to communicate using HTTP receiver SSL. The security of data transmission is managed by encrypting information and providing authentication with HTTP receiver SSL certificates. This checkbox is not selected by default.

Note: If you select the checkbox, the event source accepts SSL connections only. Also, if you change this setting, you must stop and restart Syslog collection for the change to become effective.

Name:
Certificate
Description:
Select the name of the HTTP receiver server’s SSL certificate.

Name:
Key
Description:
Select the name of the HTTP receiver server’s SSL key.

IPFIX Event Source Parameters

The following table lists the IPFIX event source parameters.

Name:
Name *
Description:
Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description:
Select the checkbox to enable the event source configuration to start collection. The checkbox is selected by default.

Name:
Description
Description:
Enter a text description for the event source.

Name: Port Number
Description: Enter the port number that you configured for your event source. The default value of port number is 4739.

Name:
Event Destination *
Description:
Select the NetWitness Log Collector or Log Decoder to which event needs to be sent from the drop-down list.

Name:
Test Configuration
Description:
Checks the configuration parameters specified in this dialog to ensure they are correct.

Kubernetes Event Source Parameters

The following table lists the Kubernetes receiver event source parameters.

Name:
Name *
Description: Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description: Select the checkbox to enable the event source configuration to start collection. The checkbox is selected by default.

Name:
Description
Description: Enter a text description for the event source.

Name:
Port Number*
Description: Enter the port number that you configured for your event sources. The default value of port number is 5044.

Name:
Event Destination*
Description:
Select the NetWitness Log Collector or Log Decoder to which event needs to be sent from the drop-down list.

Name:
Test Configuration
Description:
Checks the configuration parameters specified in this dialog to ensure they are correct.

JDBC Oracle 11g Auditing Event Source Parameters

The following table lists the JDBC Oracle 11g auditing event source parameters.

Name:
Name *
Description: Enter an alpha-numeric, descriptive name for the source. This value is only used for displaying the name on this screen.

Name:
Enabled
Description: Select the checkbox to enable the event source configuration to start collection. The checkbox is selected by default.

Name:
Description
Description: Enter a text description for the event source.

Name: Host ID*
Description: Enter the IP address of the machine where the oracle 11g database server is installed.

Name:
Port Number*
Description: Enter the port number that you configured for your event sources. The default value of port number is 1521.

Name: Database Name*
Description: Enter the name of the database where the audit tables exists.

Name:
User ID*
Description:
Enter the username of oracle 11g database.

Name: Password*
Description: Enter the password to log into the oracle 11g database.

Name:

Name: ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the pipeline will pull the data from the database.,,,,,,, If the polling interval is 1, then the pipeline will pull the data from the database for every 1 minute. If the polling interval is 2, then the pipeline will pull the data from the database for every 2 minute. This filed takes the values between 1 to 60.,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, descriptive name for the source. This value is only used for displaying the name on this screen.

Name: ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the pipeline will pull the data from the database.,,,,,,, If the polling interval is 1, then the pipeline will pull the data from the database for every 1 minute. If the polling interval is 2, then the pipeline will pull the data from the database for every 2 minute. This filed takes the values between 1 to 60.,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, descriptive name for the source. This value is only used for displaying the name on this screen.

Name: ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the pipeline will pull the data from the database.,,,,,,, If the polling interval is 1, then the pipeline will pull the data from the database for every 1 minute. If the polling interval is 2, then the pipeline will pull the data from the database for every 2 minute. This filed takes the values between 1 to 60.,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, descriptive name for the source. This value is only used for displaying the name on this screen.

Name: ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the pipeline will pull the data from the database.,,,,,,, If the polling interval is 1, then the pipeline will pull the data from the database for every 1 minute. If the polling interval is 2, then the pipeline will pull the data from the database for every 2 minute. This filed takes the values between 1 to 60.,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, descriptive name for the source. This value is only used for displaying the name on this screen.

Name: ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the pipeline will pull the data from the database.,,,,,,, If the polling interval is 1, then the pipeline will pull the data from the database for every 1 minute. If the polling interval is 2, then the pipeline will pull the data from the database for every 2 minute. This filed takes the values between 1 to 60.,,,,, the text area field will also supports the new custom query.

Name: ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, if necessary.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the change takes effect immediately (no restart required). The debug logging is verbose, so limit the number of event sources to minimize performance impact.,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, which is used during SSL connection. /etc/logstash/pki is a path in the Log Collector node.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, which is used during SSL connection. /etc/logstash/pki is a path in the Log Collector node.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, did and sessionid are collected in addition to the metas you added for aggregation.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, select * where user.dst = 'john'.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, you must provide the Elastic host, username, and password.,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, automatic key is generated and stored in the key store management for the password set in Logstash pipeline configuration. You can view the key instead of password in the Elastic Password field.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, if the value is set to 30 minutes, Log Collector starts collecting the logs and metas starting from last 30 minutes.,,,,,, this option is not enabled.

Name: ,,,,,,, ,,,,,,, ,,,,,,, in case you have multiple inputs or another set of outputs to send somewhere in addition to a NetWitness Log Collector or Log Decoder.,,,,,,, you can configure the data to be sent to Elasticsearch. In this case each event that is sent to Netwitness Platform will also be send to Elasticsearch.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, you must install the required plugin, for more information, see Install or Manage Logstash Plugin.,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, 5000 or UDP:5000, TCP:5000), and ensure the checkbox is checked. This allows the plugins to collect logs over the network (For example, UDP, TCP).,,, ensure you provide beats event source port (For example, 5044) in the advance configuration even if you have updated the port in the basic parameters.,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, select * where user.dst = 'john'.,,,,,,, http_receiver typespec, ipfix typespec, kubernetes typespec, jdbc_oracle_11g_auditing_typespec, jdbc_oracle_18c_auditing, jdbc_oracle_19c_auditing typespec, ibmdb2 typespec and jdbc_custom typespec)
Description: Select the checkbox to communicate using destination SSL.

Name: Custom SQL Statement (This field is applicable only for jdbc_oracle_18c_auditing typespec, jdbc_oracle_19c_auditing typespec ,,,,,,, ,,,,,,, Log Collector will send events to Log Decoder embedded in JSON encoded Logstash events. When not selected, Log Collector will send events in the collected format.,,,,,, ,,,,,,, see "New Health and Wellness Dashboards" topic in the System Maintenance Guide.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, LD1) which collects logstash events, from the drop-down list.,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, a custom parser or Log Parser Rules are required for full parsing.,,,,,,, ,,,,,,, ,,,,,,, Logstash related plugins are installed when Logstash is installed. In addition, you can add or customize the plugins based on your requirement.,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, see the following command:,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, select > View > Config to display the Log Collection configuration parameter tabs.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, select Logstash > Keystore management from the drop-down menu.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, enter the name of the key.,,,,,,, ,,,,,,, enter the password.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,,