Netwitness Log Decoder's rabbitmq queue is backlogged with the error Failed to locate message id field

Issue

The Log Decoder's rabbitmq queues show a huge backlog when running the below command.

#rabbitmqctl list_queues -p logcollection consumers messages name

In the /var/log/messages file a similar error as below will be indicated.
Jul 11 16:49:35 Logdecoder NwLogCollector[1788]: [GenericLogTransformer] [warning] Failed to locate message id field during event transformation. The message id field number is 5. The raw event is "0x7fd,6ca,956,1e0". The message will still be delivered.

Cause

This issue is due to incorrectly formatted events from syslog or file collection sources. These incorrectly formatted events are identified by checking the  Events page using a device.type='unknown'  query to see the below problematic events.

Resolution

Work with the event source owner to stop or fix the problematic events. The device the event is coming from can be identified by the device.ip meta associated with the event.

Alternatively, Syslog or File Collection filters can be used to drop problematic event collections using the below document.
Configure Event Filters for a Collector

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Log Decoder, RabbitMQ Message Broker
RSA Version/Condition: 11.x,12.X
Platform: CentOS 7

Summary

This document outlines the procedure to stop problematic event collection.

Approval Reviewer Queue

Technical approval queue