Log Parser Tool v2 User Guide

The following article contains a summary of the NetWitness® Log Parser Tool version 2 - User Guide. To see the full guide, go to Attachments on this article and download the associated PDF.
Click the following link to download the NetWitness® Log Parser Tool version 2.

Summary of the NetWitness® Log Parser Tool Version 2 - User Guide

Purpose

The NetWitness® Log Parser Tool v2 (LPT) is a graphical utility for creating, editing, and customizing log parsers that run on the NetWitness® Log Decoder. It enables users to define how logs from various event sources are identified, parsed, and how metadata is extracted for security analysis and reporting

Key Features & Concepts

• Graphical & XML Editing: Users can build parsers visually or directly edit the underlying XML for advanced customization.
• Parser Structure:
  • Base XML: Contains standard parser definitions.
  • Custom XML: Stores user customizations, ensuring updates from NetWitness® don’t overwrite changes.
  • Tokens XML: Used internally for accurate parsing replication but not directly edited by users
• Terminology Updates:
  • Discovery Patterns: Formerly HEADER elements, used for device/event identification.
  • Message Patterns: Formerly MESSAGE elements, used for parsing event payloads.
  • Format Types and Typed Variables: For advanced data extraction and validation.
  • Parse Rules: Define static tokens for targeted parsing.

Workflow Overview

1. Obtain Log Files:
   Gather representative logs from the event source, ensuring all unique event types are included.
2. Create or Edit Parsers:
   • Define Discovery Patterns (headers) to identify event sources.
   • Define Message Patterns to extract variable data from event payloads.
   • Assign meta mappings to extracted variables for NetWitness analysis.
3. Validate and Test:
   • Use the Samples Messages panel to test parsing accuracy.
   • Status indicators (green/yellow/red) show match completeness.
4. Export and Deploy:
   • Save individual XML files or export as a parser package (.envision format) for deployment to Log Decoders.
5. Import/Export Table Maps:
   • Manage mappings between parser variables and NetWitness® meta keys for consistent metadata extraction.

Advanced Capabilities

• Type Parsing:
  Use Format Types and Types Variables for complex data (e.g. JSON, Base64, IP addresses, and domains).
• Variants and Inheritance:
  Support for multiple formats per variable (e.g. IPv4, IPv6, and hostname) and inheritance for reusable parsing logic.
• Parse Rules:
  Post-session scanning for extracting data from logs that don't match defined patterns.
• TAGVAL (Key-Value) Parsing:
  Easily parse logs with named value pairs using configurable delimiters.
• CEF Parser Customization:
  Special support for Common Event Format (CEF) logs, including vendor/product mapping and field overrides.

User Experience

• UI Controls:
  • Dark/Light theme support
  • Undo/Redo for parser edits
  • Import/export for both parsers and table maps
  • Drag-and-drop and context menu for pattern and mapping management
• Error Handling and Validation:
  • Visual feedback for incomplete or incorrect patterns.
  • Guidance for precedence and specificity in pattern matching.

Best Practices and Notes

• Always order Discovery and Message Patterns from most specific to most generic to avoid misidentification.
• Use custom XML for modifications to avoid conflicts with NetWitness updates.
• Validate parsing with sample logs before deployment.
• Use Table Maps to ensure extracted variables are mapped to the correct NetWitness meta keys.
• For CEF and other standardized formats, leverage built-in mappings and extend as needed.

The following article contains a summary of the NetWitness® Log Parser Tool version 2 - User Guide. To see the full guide, go to Attachments on this article and download the associated PDF.
Click the following link to download the NetWitness® Log Parser Tool version 2.