Fix an issue with RabbitMQ lacking consumers on log queues after upgrading to 11.5.3 leading to log collection issues

Issue

1. Missing logs in Investigate
2. H&W Alerts for "LogCollector Event Processor Queue with No Consumer" or similar
3. H&W Alerts for "Critical RabbitMQ Queue Message Count"
4. H&W Alerts for disk space due to a buildup of RDQ files on the Collectors

Cause

This behavior has been observed on upgrades to 11.5.3 and potentially 11.5.2 and may be attributable to a corrupt nw_admin.ez plug-in.

Workaround

1. Rabbitmq shovels show green on either the RLC (push) or the Local LC (Pull) - Green indicates that the shovel is up and running, but it does not highlight missing consumers.
2. Running the following commands on the Remote Log Collector and the Local Log Collector to show one or more Event Queues with no consumers:

• Download the provided nw_admin-11.5.3.0.ez
• SCP it to /root/ on all Log Collectors (remote and local)
• Run a sha256sum on it to confirm it is the correct version and was not corrupted in transit.

• Stop rabbitmq-server: systemctl stop rabbitmq-server
• Take a backup of the current nw_admin.ez plug-in currently on the filesystem:
  • mv /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins/nw_admin.ez /root/nw_admin.ez.BAK
• Copy the new version to the appropriate location (taking special consideration to rename it to simply "nw_admin.ez"):
  • cp /root/nw_admin-11.5.3.0.ez /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/plugins/nw_admin.ez
• Change the permissions:

• Verify the sha256sum and permissions once more:

•  Make a backup of the original plug-in and also copy the new plug-in to our chef reference directory for potential future use:

• Verify the sha256sum of the backup and patched version once again:

• Start rabbitmq-server: systemctl start rabbitmq-server
• Restart the nwlogcollector service: systemctl restart nwlogcollector
  • Do this on both the local and remote LCs.
• Verify that rabbitmq-server now recognizes the patched version (should be showing as "11.5.1.0") when doing a "rabbitmq-plugins list".

• Then verify that the consumers have returned:

•  After 15 minutes, check that normal log flow in the UI has returned per lc.cid or did not.

1. Stop rabbitmq and nwlogcollector services from "services.msc"
2. Backup and remove the existing nw_admin pluggin from below 2 locations. (note: there may be previous versions in addition to 11.5.3.0, remove all nw_admin plugins in the following directories):
   1. C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.3\plugins
   2. C:\Program Files\NwLogCollector
3. Copy the nw_admin.ez plug-in attached on this KB (that is also used on the CentOS appliances) to the following directories:
   1. C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.3\plugins
   2. C:\Program Files\NwLogCollector
4. Start rabbitmq and nwlogcollector services from "services.msc"

Resolution

To install a patched version of the nw_admin.ez plug-in and restart the appropriate services.

Internal Comments

• https://bedfordjira.na.rsa.net/browse/SACE-15832
• https://bedfordjira.na.rsa.net/browse/ASOC-109851

Product Details

Summary

After upgrading to 11.5.3, logs flowing from remote to local collectors may unexpectedly quit working. When using rabbitmqctl list_queues, you would notice certain (but not necessarily all) consumers are missing. This occurs in both push and pull environments.

Approval Reviewer Queue

Technical approval queue