.png?width=180&height=32&name=NW%20Logo%20(2).png){kind=small}
Archiver in initialization state and aggregation is not starting in RSA NetWitness Logs and Network
Issue
Archiver service in initialization state and aggregation is not starting.Further checking, Archiver service appears to have an issue with one of its collections - in this example 'ACVR11WINDOWS'
[root@CHEETAHACVR11 ACVR11WINDOWS]# tail -n10000 /var/log/messages | grep NwArchiver | grep fail
May 31 16:04:43 CHEETAHACVR11 NwArchiver[25890]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker
May 31 16:04:43 CHEETAHACVR11 NwArchiver[25890]: [Engine] [failure] Module archiver failed to load: Diagnostic information: Throw in function virtual nw::AggStatePtr nw::CollectionBroker::addAggState(nw::AggStatePtr, bool)Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker[boost::errinfo_at_line_*] = 99
May 31 16:04:45 CHEETAHACVR11 NwArchiver[25890]: [Broker] [failure] Internal collection {ACVR11WINDOWS} is in a failed state
May 31 16:04:45 CHEETAHACVR11 NwArchiver[25890]: [Broker] [failure] Throw in function virtual void nw::CollectionBroker::online()Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} is in a failed state[boost::errinfo_at_line_*] = 123
Jun 1 03:44:03 CHEETAHACVR11 collectd[3610]: NgNativeReader_NwArchiver-FastUpdate: failed to connect to nws://admin@localhost:56008/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2Fed1d6d0e-7e90-487a-a3f6-95468cb2642b.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2Fed1d6d0e-7e90-487a-a3f6-95468cb2642b.pem
Jun 1 03:44:03 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] SNMP AgentX Master connection is DOWN due to: No such file or directory. Likely cause: snmpd is disabled or misconfigured.
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] Throw in function virtual void nw::BrokerState::load()Dynamic exception type: boost::exception_detail::clone_impl<nw::LogicError>std::exception::what: There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct[boost::errinfo_at_line_*] = 1134
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [Aggregation] [failure] Failed to initialize device '{ACVR11WINDOWS}' because There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct. Device aggregation is being stopped.
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Diagnostic information: Throw in function virtual nw::AggStatePtr nw::CollectionBroker::addAggState(nw::AggStatePtr, bool)Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker[boost::errinfo_at_line_*] = 99
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Broker] [failure] Internal collection {ACVR11WINDOWS} is in a failed state
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Broker] [failure] Throw in function virtual void nw::CollectionBroker::onlin
Jun 1 08:03:49 CHEETAHACVR11 NwArchiver[11966]: [{ACVR11WINDOWS}] [failure] There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct
May 31 16:04:43 CHEETAHACVR11 NwArchiver[25890]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker
May 31 16:04:43 CHEETAHACVR11 NwArchiver[25890]: [Engine] [failure] Module archiver failed to load: Diagnostic information: Throw in function virtual nw::AggStatePtr nw::CollectionBroker::addAggState(nw::AggStatePtr, bool)Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker[boost::errinfo_at_line_*] = 99
May 31 16:04:45 CHEETAHACVR11 NwArchiver[25890]: [Broker] [failure] Internal collection {ACVR11WINDOWS} is in a failed state
May 31 16:04:45 CHEETAHACVR11 NwArchiver[25890]: [Broker] [failure] Throw in function virtual void nw::CollectionBroker::online()Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} is in a failed state[boost::errinfo_at_line_*] = 123
Jun 1 03:44:03 CHEETAHACVR11 collectd[3610]: NgNativeReader_NwArchiver-FastUpdate: failed to connect to nws://admin@localhost:56008/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2Fed1d6d0e-7e90-487a-a3f6-95468cb2642b.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2Fed1d6d0e-7e90-487a-a3f6-95468cb2642b.pem
Jun 1 03:44:03 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] SNMP AgentX Master connection is DOWN due to: No such file or directory. Likely cause: snmpd is disabled or misconfigured.
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] Throw in function virtual void nw::BrokerState::load()Dynamic exception type: boost::exception_detail::clone_impl<nw::LogicError>std::exception::what: There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct[boost::errinfo_at_line_*] = 1134
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [Aggregation] [failure] Failed to initialize device '{ACVR11WINDOWS}' because There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct. Device aggregation is being stopped.
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Diagnostic information: Throw in function virtual nw::AggStatePtr nw::CollectionBroker::addAggState(nw::AggStatePtr, bool)Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker[boost::errinfo_at_line_*] = 99
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Broker] [failure] Internal collection {ACVR11WINDOWS} is in a failed state
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Broker] [failure] Throw in function virtual void nw::CollectionBroker::onlin
Jun 1 08:03:49 CHEETAHACVR11 NwArchiver[11966]: [{ACVR11WINDOWS}] [failure] There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct
Cause
As seen from the error messages above, there seems to be some mappings problem on the collection;
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] Throw in function virtual void nw::BrokerState::load()Dynamic exception type: boost::exception_detail::clone_impl<nw::LogicError>std::exception::what: There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct[boost::errinfo_at_line_*] = 1134
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [Aggregation] [failure] Failed to initialize device '{ACVR11WINDOWS}' because There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct. Device aggregation is being stopped.
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [Aggregation] [failure] Failed to initialize device '{ACVR11WINDOWS}' because There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct. Device aggregation is being stopped.
Looks like mappings between lSession and vSession corrupted when root user initiated force shutdown by command and update could not persist. Archiver session will usually detect these errors and does not allow aggregation to start until they are fixed.
Resolution
1. Determine the problematic collection, here it is ACVR11WINDOWS.
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker
2. SSH to Archiver service and shut down Archiver service.
[root@Archiver ~]# systemctl stop nwarchiver
3. Navigate to the Archiver's index folder, usually /var/netwitness/archiver/index. and
NOT specific collection's index folder (such as /var/netwitness/archiver/ACVR11WINDOWS/index)
4. Move the collection's index/mapping files to another directory ex. /tmp/windows/
mv ACVR11WINDOWS-* /tmp/windows/
5. Start Archiver service
[root@Archiver ~]# systemctl start nwarchiver
Product Details
RSA Product Set: NetWitness Logs and NetworkRSA Product/Service Type: Archiver
RSA Version/Condition: 10.6.X
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue