Skip to content
  • There are no suggestions because the search field is empty.

NetWitness nwbroker.service is unable to start in NetWitness Platform

Issue

NwBroker service was failing to start as shown below in NW
  • Column 1: # systemctl status nwbroker 
       nwbroker.service - Netwitness Broker
       Loaded: loaded (/usr/lib/systemd/system/nwbroker.service; enabled; vendor preset: disabled)
       Active: failed (Result: start-limit) since Wed  15:20:36 KST; 24h ago
     Main PID: 259920 (code=exited, status=1/FAILURE)

    Aug 21 15:20:35 mss-broker1 systemd[1]: Unit nwbroker.service entered failed state.
    Aug 21 15:20:35 mss-broker1 systemd[1]: nwbroker.service failed.
    Aug 21 15:20:35 mss-broker1 systemd[1]: nwbroker.service holdoff time over, scheduling restart.
    Aug 21 15:20:35 mss-broker1 systemd[1]: start request repeated too quickly for nwbroker.service
    Aug 21 15:20:35 mss-broker1 systemd[1]: Failed to start Netwitness Broker.
    Aug 21 15:20:35 mss-broker1 systemd[1]: Unit nwbroker.service entered failed state.
    Aug 21 15:20:35 mss-broker1 systemd[1]: nwbroker.service failed.
    Warning: nwbroker.service changed on disk. Run 'systemctl daemon-reload' to reload units.

 
  • Column 1: # /usr/sbin/NwBroker
    (i) 2019-Aug-22 15:49:45 [Engine]  RSA NetWitness Service Copyright 2001-2019, RSA Security Inc. All Rights Reserved.
    (i) 2019-Aug-22 15:49:45 [Engine]  Running broker in console
    (d) 2019-Aug-22 15:49:45 [Engine]  [broker](7f1318d5d940): Entering ServiceBase::Initialize()
    (d) 2019-Aug-22 15:49:45 [Engine]  [broker](7f1318d5d940): ServiceBase::SetStatus(Stopped, Start Pending)
    (a) 2019-Aug-22 15:49:45 [Engine]  RSA NetWitness Service, Broker 11.3.1.0 (Jun 14 2019) 64 bit Starting
    (F) 2019-Aug-22 15:49:45 [Engine]  Failed to start engine because of exception: Throw in function X509* nw::{anonymous}::getX509FromPEM(const boost::filesystem::path&)
    Dynamic exception type: boost::exception_detail::clone_impl
    std::exception::what: error loading trusted certificate file
    [nw::ssl_error_tag*] = error:0E06D06C:configuration file routines:NCONF_get_string:no value error:02001002:system library:fopen:No such file or directory error:2006D080:BIO routines:BIO_new_file:no such file
    [boost::errinfo_at_line_*] = 45
    [boost::errinfo_file_name_*] = /etc/netwitness/ng/broker/trustpeers/c5al34bl.0
    [boost::errinfo_api_function_*] = BIO_new_file

 
  • Column 1: #ls -al /etc/netwitness/ng/broker/trustpeers/ <-- You need to note this information for restoration after pem file regeneration
    total 0
    drwxr-x---. 2 netwitness netwitness 78 May  2 05:11 .
    drwxr-x---. 6 netwitness netwitness 95 May  2 05:10 ..
    lrwxrwxrwx. 1 root       root       72 May  2 05:11 1386a7d5.0 -> /etc/pki/nw/peer/respond-server/12b060c9-6f99-4356-8db5-e00313277604.pem
    lrwxrwxrwx. 1 root       root       40 May  2 05:10 67342faa.0 -> /etc/pki/nw/nextgen/nwappliance-cert.pem
    lrwxrwxrwx. 1 root       root       30 May  2 05:10 90204291.0 -> /etc/pki/nw/node/node-cert.pem
    lrwxrwxrwx. 1 root       root       67 May  2 05:10 cf280d67.0 -> /etc/pki/nw/peer/sa-server/b311eddf-8142-46bd-b801-9b80afda3dfe.pem


Cause

For some reason, the pem certificate file for the sa-server service id was broken or missing in /etc/pki/nw/peer/sa-server.
In this case, nwbroker service is unable to start.

Resolution

You need to re-generate the pem file for the sa-server service id in this case.

Please follow the steps below.
  1. Get ssl certificate information and save it to file('root.out')
    • Column 1: #openssl s_client -connect localhost:7000 -tls1_2 > root.out
      depth=2 C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness Platform, CN = NetWitness Root CA
      verify return:1
      depth=1 CN = NetWitness Intermediate CA, OU = NetWitness, O = RSA, L = Reston, ST = VA, C = US
      verify return:1
      depth=0 C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = e431bf77-a43c-4a8a-b9cf-4ecd1b34a7c4
      verify return:1


     
  2. Edit it using vi, then extract content and save it as a b311eddf-8142-46bd-b801-9b80afda3dfe.pem file.
    Note: certificate is the copy of the section:

    -----BEGIN CERTIFICATE-----
    to
    -----END CERTIFICATE-----

    • Column 1: #vi b311eddf-8142-46bd-b801-9b80afda3dfe.pem

  3. Locate pem file into /etc/pki/nw/peer/sa-server/ and link it same as before.
     
    • Column 1: #ls -al /etc/netwitness/ng/broker/trustpeers
      total 0
      drwxr-x---. 2 netwitness netwitness 78 May  2 05:11 .
      drwxr-x---. 6 netwitness netwitness 95 May  2 05:10 ..
      lrwxrwxrwx. 1 root       root       72 May  2 05:11 1386a7d5.0 -> /etc/pki/nw/peer/respond-server/12b060c9-6f99-4356-8db5-e00313277604.pem
      lrwxrwxrwx. 1 root       root       40 May  2 05:10 67342faa.0 -> /etc/pki/nw/nextgen/nwappliance-cert.pem
      lrwxrwxrwx. 1 root       root       30 May  2 05:10 90204291.0 -> /etc/pki/nw/node/node-cert.pem
      lrwxrwxrwx. 1 root       root       67 May  2 05:10 cf280d67.0 -> /etc/pki/nw/peer/sa-server/b311eddf-8142-46bd-b801-9b80afda3dfe.pem


    Once completed, you are able to start nwbroker.service without any issue.

Product Details

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Concentrator
NetWitness Version/Condition: 11.x /12.X
Platform: CentOS 7 / Alma

Summary

NwBroker service was failing to start due to broken or missing pem certificate file.


Approval Reviewer Queue

Technical approval queue