RSA Security Analytics Archiver cannot be started due to additional files in the trustpeers folder

Issue

The Archiver service cannot be started within the Security Analytics UI.

Messages similar to the following are seen in the /var/log/messages file when the service is started manually from the command line with the start nwarchiver command:

 Jan 20 17:21:49 RSAARCHIVER init: nwarchiver main process (31161) terminated with status 1
 
Jan 20 17:21:49 RSAARCHIVER init: nwarchiver main process ended, respawning

Running the NwArchiver executable results in the following error:

 [root@RSAARCHIVER ng]# /usr/sbin/NwArchiver
 
(i) 2015-Jan-20 17:27:38 [Engine] RSA Security Analytics Service Copyright 2001-2014, RSA Security Inc. All Rights Reserved.
 
(i) 2015-Jan-20 17:27:38 [Engine] Running archiver in console
 
(d) 2015-Jan-20 17:27:38 [Engine] [archiver](7f420d417800): Entering ServiceBase::Initialize()
 
(d) 2015-Jan-20 17:27:38 [Engine] [archiver](7f420d417800): ServiceBase::SetStatus(Stopped, Start Pending)
 
(i) 2015-Jan-20 17:27:38 [Engine] RSA Security Analytics Service, Archiver 10.4.0.2.3360 (Oct 9 2014) 64 bit Starting
 
(F) 2015-Jan-20 17:27:38 [Engine] Failed to start engine because of exception: Throw in function X509* nw::{anonymous}::getX509FromPEM(const boost::filesystem::path&)
 
Dynamic exception type: N5boost16exception_detail10clone_implIN2nw9ExceptionEEE
 
std::exception::what: error parsing certificate file
 
[PN2nw13ssl_error_tagE] = error:0906D06C:PEM routines:PEM_read_bio:no start line
 
[PN5boost16errinfo_at_line_E] = 56
 
[PN5boost18errinfo_file_name_E] = /etc/netwitness/ng/archiver/trustpeers/CmdTool.log
 
[PN5boost21errinfo_api_function_E] = PEM_read_bio_X509

Cause

This issue occurs because additional (unwanted) files exist in the /etc/netwitness/ng/archiver/trustpeers/ folder which cause the Archiver service to read those files as certificates.

Resolution

In order to resolve the issue, remove the offending file(s) in /etc/netwitness/ng/archiver/trustpeers/ and then start the Archiver service again.

The example below shows that CmdTool.log and MegaSAS.log are in the trustpeers folder (most likely as a result of running the nwraidutil.pl script from that folder).
Once they are removed, Archiver is able to start normally.

 [root@RSAARCHIVER archiver]# cd /etc/netwitness/ng/archiver/trustpeers/
 
[root@RSAARCHIVER trustpeers]# ll
 
total 92
 
-rw-r--r--. 1 root root 2102 Jan 2 15:34 3cda430d.0
 
-rw-r--r--. 1 root root 2009 Jan 2 15:34 b33fd481.0
 
-rw-r--r--. 1 root root 300 Jan 2 15:42 CmdTool.log
 
-rw-r--r--. 1 root root 79477 Jan 2 15:42 MegaSAS.log
 
[root@RSAARCHIVER trustpeers]# rm -f CmdTool.log
 
[root@RSAARCHIVER trustpeers]# status nwarchiver
 
nwarchiver start/running
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]# stop nwarchiver
 
nwarchiver stop/waiting
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]# start nwarchiver && tail -f /var/log/messages |grep -v collectd
 
nwarchiver start/running
 
Jan 21 10:10:03 RSAARCHIVER init: nwarchiver main process (22598) terminated with status 1
 
Jan 21 10:10:03 RSAARCHIVER init: nwarchiver main process ended, respawning
 
^C
 
[root@RSAARCHIVER trustpeers]# ll
 
total 88
 
-rw-r--r--. 1 root root 2102 Jan 2 15:34 3cda430d.0
 
-rw-r--r--. 1 root root 2009 Jan 2 15:34 b33fd481.0
 
-rw-r--r--. 1 root root 79477 Jan 2 15:42 MegaSAS.log
 
[root@RSAARCHIVER trustpeers]# stop nwarchiver
 
nwarchiver stop/waiting
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]# /usr/sbin/NwArchiver
 
(i) 2015-Jan-21 10:10:18 [Engine] RSA Security Analytics Service Copyright 2001-2014, RSA Security Inc. All Rights Reserved.
 
(i) 2015-Jan-21 10:10:18 [Engine] Running archiver in console
 
(d) 2015-Jan-21 10:10:18 [Engine] [archiver](7fcd59b62800): Entering ServiceBase::Initialize()
 
(d) 2015-Jan-21 10:10:18 [Engine] [archiver](7fcd59b62800): ServiceBase::SetStatus(Stopped, Start Pending)
 
(i) 2015-Jan-21 10:10:18 [Engine] RSA Security Analytics Service, Archiver 10.4.0.2.3360 (Oct 9 2014) 64 bit Starting
 
(F) 2015-Jan-21 10:10:18 [Engine] Failed to start engine because of exception: Throw in function X509* nw::{anonymous}::getX509FromPEM(const boost::filesystem::path&)
 
Dynamic exception type: N5boost16exception_detail10clone_implIN2nw9ExceptionEEE
 
std::exception::what: error parsing certificate file
 
[PN2nw13ssl_error_tagE] = error:0906D06C:PEM routines:PEM_read_bio:no start line
 
[PN5boost16errinfo_at_line_E] = 56
 
[PN5boost18errinfo_file_name_E] = /etc/netwitness/ng/archiver/trustpeers/MegaSAS.log
 
[PN5boost21errinfo_api_function_E] = PEM_read_bio_X509
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]# ll
 
total 88
 
-rw-r--r--. 1 root root 2102 Jan 2 15:34 3cda430d.0
 
-rw-r--r--. 1 root root 2009 Jan 2 15:34 b33fd481.0
 
-rw-r--r--. 1 root root 79477 Jan 2 15:42 MegaSAS.log
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]#
 
[root@RSAARCHIVER trustpeers]# rm -f MegaSAS.log
 
[root@RSAARCHIVER trustpeers]# status nwarchiver
 
nwarchiver stop/waiting
 
[root@RSAARCHIVER trustpeers]# ll
 
total 8
 
-rw-r--r--. 1 root root 2102 Jan 2 15:34 3cda430d.0
 
-rw-r--r--. 1 root root 2009 Jan 2 15:34 b33fd481.0
 
[root@RSAARCHIVER trustpeers]# start nwarchiver && tail -f /var/log/messages |grep -v collectd
 
nwarchiver start/running, process 22641
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] RSA Security Analytics Service Copyright 2001-2014, RSA Security Inc. All Rights Reserved.
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] Running archiver in console
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] RSA Security Analytics Service, Archiver 10.4.0.2.3360 (Oct 9 2014) 64 bit Starting
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] Configuration loaded from /etc/netwitness/ng/NwArchiver.cfg
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] Initializing OpenSSL 1.0.0-fips 29 Mar 2010
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] Creating a pool of 20 server threads
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] Loading module 'archiver'
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Thread] [info] Starting thread: Engine Stats id: 22642
 
Jan 21 10:10:38 RSAARCHIVER nw[22641]: [Engine] [info] Security Analytics Archiver Server 'RSAARCHIVER' is running and listening on port 50008 and SSL port 56008
 
Jan 21 10:10:38 RSAARCHIVER nw[31169]: [Appliance] [info] archiver started on port 50008
 
...
 
...
 
Jan 21 10:11:04 RSAARCHIVER nw[22641]: [Rest] [info] REST service listening on port 50108

If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.

Product Details

RSA Product Set: Security Analytics
RSA Product/Service Type: Archiver
RSA Version/Condition: 10.4 and above
Platform: CentOS
O/S Version: EL6

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue