Skip to content
  • There are no suggestions because the search field is empty.

RSA NetWitness Logdecoder Version Update failing due to rabbitmq service stopped

Issue

Netwitness Logdecoder version Update fails as rabbitmq service stopped with below errors.

/var/log/rabbitmq/rabbitmq@ .log:
2021-03-04 06:57:09.207 [error] <0.8.0>
Error description:
init:do_boot/3
init:start_em/1
rabbit:start_it/1 line 484
rabbit:'-boot/0-fun-0-'/0 line 333
rabbit_upgrade:maybe_upgrade_mnesia/0 line 142
rabbit_upgrade:ensure_backup_taken/0 line 98
throw:{error,previous_upgrade_failed}
Log file(s) (may contain more information):
/var/log/rabbitmq/rabbit@577719d8-d61e-428c-b2f4-f8d494b334ae.log
/var/log/rabbitmq/rabbit@577719d8-d61e-428c-b2f4-f8d494b334ae_upgrade.log
2021-03-04 06:57:23.017 [info] <0.8.0> Log file opened with Lager
2021-03-04 06:57:25.082 [info] <0.8.0> Feature flags: list of feature flags found:
2021-03-04 06:57:25.082 [info] <0.8.0> Feature flags: [ ] drop_unroutable_metric
2021-03-04 06:57:25.082 [info] <0.8.0> Feature flags: [ ] empty_basic_get_metric
2021-03-04 06:57:25.082 [info] <0.8.0> Feature flags: [ ] implicit_default_bindings
2021-03-04 06:57:25.082 [info] <0.8.0> Feature flags: [ ] quorum_queue
2021-03-04 06:57:25.082 [info] <0.8.0> Feature flags: [ ] virtual_host_metadata
2021-03-04 06:57:25.083 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2021-03-04 06:57:25.149 [error] <0.8.0> Found lock file at /var/netwitness/rabbitmq/mnesia/rabbit@577719d8-d61e-428c-b2f4-f8d494b334ae/schema_upgrade_lock.
Either previous upgrade is in progress or has failed.
Database backup path: /var/netwitness/rabbitmq/mnesia/rabbit@577719d8-d61e-428c-b2f4-f8d494b334ae-upgrade-backup

Cause

During the upgrade, the rabbitmq was backed up with RDQ and use a high amount of memory. Since it had no additional memory to spare, it failed to upgrade itself and left rabbitmq corrupted state.


Resolution

Please follow the below steps to start rabbitmq service.
  • Please stop logcollection using the below command.
systemctl stop nwlogcollector
  • Please stop Capture in Logdecoder->System page.
  • Please take backup of staled rdq files to different directory where enough space is available.
cd /var/netwitness/rabbitmq/mnesia/rabbit\@ /msg_store_persistent/
mv * /var/netwitness/rdqbackup

Note: Please use below existing KB for reinjecting these backed up rdq files.
Reprocessing Orphaned RDQ files for RSA Netwitness Log Decoders/Log Collectors
  • Take backup of mnesia directory using below command.
cd /var/netwitness/rabbitmq/
mv mnesia /var/netwitness/mnesiabackup
  • Then start rabbitmq service using the below command and check status.
service rabbitmq-server start

service rabbitmq-server status
  • Then initiate the update which will go successful.

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.5.1.0
Platform: CentOS
O/S Version: 7

Summary

This document outlines the procedure to start rabbitmq service to complete version update.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue