LogStash Integration Guide for 12.5.1

Overview

This section provides an overview of how Logstash integrates with NetWitness® Platform version 12.5.1. It explains the purpose of the integration and outlines the primary use cases, including supporting event sources that do not have native NetWitness® integrations and enabling customers with existing Logstash deployments to forward logs to NetWitness® with minimal changes. The section also introduces the concept of Managed Logstash, where Logstash is packaged and supported as part of the NetWitness® Log Collector or Virtual Log Collector, eliminating the need for an external Logstash deployment. A high-level architectural diagram illustrates how event sources, Logstash, and NetWitness® components interact to process and decode logs.

Configuration Process

This section explains the end-to-end configuration workflow required to integrate Logstash with NetWitness®. It walks through the decision-based flow that determines whether Logstash, the NetWitness® codec, event source inputs, enrichment filters, and JSON parsers need to be installed or configured. It also describes the logical data flow from the moment an event is generated, through collection and processing in Logstash, encoding via the NetWitness® codec, transmission to a Log Decoder or Virtual Log Collector, and final meta population using a JSON parser. This section establishes the conceptual foundation for the more detailed configuration steps that follow.

Install Logstash

This section explains how to install and prepare Logstash for integration with NetWitness®. It discusses supported installation options, including the open-source and Elastic distributions, and emphasizes following Logstash security and operational best practices. The section provides guidance for running Logstash as a service on Linux or Windows, enabling it to start automatically at boot, and locating Logstash logs for troubleshooting. It also highlights important considerations such as ensuring proper user permissions, especially on Linux systems, and references official Logstash documentation for installation and troubleshooting support.

Install and Configure the NetWitness® Codec

This section explains how to install and configure the NetWitness® Logstash codec, which is required to format Logstash events into RFC‑5424 syslog messages consumable by NetWitness®. It describes downloading the offline codec installer, stopping the Logstash service, removing any existing codec versions, and installing the updated package. The section also identifies the default configuration directories for Logstash input, filter, and output files and explains how to restart Logstash to activate the codec. This configuration ensures that processed events are correctly encoded before being forwarded to NetWitness®.

Configure Logstash Output Plugins

This section covers how to configure Logstash output plugins to securely forward processed events to NetWitness®. It explains using the TCP output plugin in conjunction with the NetWitness® codec and provides examples for both non-encrypted TCP communication and TLS‑encrypted communication. The section also explains how to enable certificate-based verification by creating a truststore from Log Decoder or Virtual Log Collector certificates and configuring Logstash with the appropriate CA, certificate, and private key files. These configurations ensure reliable and secure transmission of logs from Logstash to NetWitness®.

Configure the Event Source

This section provides guidance on configuring event sources that send logs to Logstash before forwarding them to NetWitness®. It explains how Filebeat can be used to collect file-based logs such as Apache logs, and how Auditbeat can be used to collect operating system audit logs, particularly from CentOS systems. The section also describes how to modify Filebeat or Auditbeat configuration files to enable desired inputs and redirect output from Elasticsearch to Logstash. This ensures that logs from different sources are consistently ingested into the Logstash pipeline.

Configure Logstash Filters to Add NetWitness® Meta

This section explains how to enrich events in Logstash with NetWitness® metadata so they can be correctly parsed and classified by the Log Decoder. It describes the required metadata fields, including the NetWitness® device type, message ID, and source host, and explains naming requirements for device types. The section also discusses optional metadata such as collection host identifiers and explains how the NetWitness® codec handles JSON payloads by default. An example filter configuration demonstrates how metadata can be conditionally added based on the event source.

Advanced NetWitness® Configuration

This section covers advanced configuration techniques for optimizing and extending the Logstash and NetWitness® integration. It explains the use of Grok filters to extract structured data, the role of input and filter plugins, and methods for filtering out unwanted logs at both the Logstash and Beats levels. The section also describes using the Heartbeat plugin to send test events, configuring persistent queues in Logstash to prevent data loss during failures, and advanced NetWitness® codec payload formatting. These configurations provide greater resilience, flexibility, and precision in log processing.

Configure NetWitness® to Collect Events

This section explains how to configure NetWitness® Log Decoders to receive and process events sent from Logstash. It describes starting or restarting capture services, verifying that capture is active, and adjusting Log Decoder settings to support larger event sizes when necessary. The section also explains how to modify decoder configuration parameters using REST endpoints and emphasizes reducing event sizes when oversized logs risk truncation. These steps ensure that NetWitness® can reliably ingest and decode incoming events.

Linux Event Source Example

This section provides a practical example of configuring Logstash to collect system and audit events from a Linux (CentOS) environment and forward them to NetWitness®. It explains sample input, filter, and output configurations using Beats plugins, illustrates how to open required firewall ports, and demonstrates how metadata is applied to distinguish different event types. The section also recommends organizing configurations into separate pipelines and shows how to define pipelines in the Logstash configuration to simplify management and scalability.

Build Custom JSON Parser

This section explains how to build a custom JSON parser for NetWitness® when default parsing is insufficient. Using Linux logs as an example, it walks through analyzing a sample JSON event, extracting required metadata from the RFC‑5424 header, mapping JSON payload fields to NetWitness® datatypes, and parsing strings, arrays, nested objects, and variable parent keys. The section demonstrates how parsed metadata appears in the Log Decoder and provides a complete example parser, illustrating advanced customization for precise metadata extraction.

Deploy JSON Parser

This section covers how to deploy and activate a custom JSON parser on a NetWitness® Log Decoder. It explains copying the parser file to the appropriate device directory, reloading parsers using REST APIs, and reloading parsers through the NetWitness® user interface. These steps ensure that newly created or updated parsers are loaded into memory and applied to incoming events without requiring a full system restart.