Skip to content
  • There are no suggestions because the search field is empty.

NetWitenss Log Collector 'Shovel Failed to Connect/Unknown CA' Errors when changing collection method from push to pull or vice-versa

Issue

"Shovel Failed to Connect/Unknown CA" Errors on NetWitness Log Collector or VLC when changing collection method from push to pull or vice-versa.

In a test where Push and Pull were configured simultaneously and then Push was removed, the errors below were displayed.
 

The following errors were logged in the /var/log/messages file on the VLC:

Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:39:56 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.39.56Z SSL: certify: ssl_connection.erl:1711:Fatal error: unknown ca
Jun 16 03:40:01 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.40.01Z error on AMQP connection <0.1117.0>: {ssl_upgrade_error,"unknown ca"}
Jun 16 03:40:01 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.40.01Z error on AMQP connection <0.1122.0>: {ssl_upgrade_error,"unknown ca"}
Jun 16 03:40:01 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.40.01Z error on AMQP connection <0.1127.0>: {ssl_upgrade_error,"unknown ca"}
Jun 16 03:40:01 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.40.01Z error on AMQP connection <0.1132.0>: {ssl_upgrade_error,"unknown ca"}
Jun 16 03:40:01 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.40.01Z error on AMQP connection <0.1137.0>: {ssl_upgrade_error,"unknown ca"}
Jun 16 03:40:01 savlc nw[2932]: [MessageBroker] [failure] error 2014-06-16T03.40.01Z error on AMQP connection <0.1157.0>: {ssl_upgrade_error,"unknown ca"}



The following errors were logged in the /var/log/messages file on the LC: 

Jun 16 03:39:40 sa nw[18214]: [MessageBroker] [failure] error 2014-06-16T03.39.40Z SSL: certify: ssl_handshake.erl:593:Fatal error: unknown ca
Jun 16 03:39:40 sa nw[18214]: [MessageBroker] [failure] error 2014-06-16T03.39.40Z Shovel failed to connect to Host: "10.44.8.73" Port: 5671 VirtualHost: <<"logcollection">>: error:{badmatch,                                                                                                   {error,                                                                                                    "unknown ca"}}
Jun 16 03:39:40 sa nw[18214]: [MessageBroker] [failure] error 2014-06-16T03.39.40Z ** Generic server <0.1062.0> terminating** Last message in was {'$gen_cast',init}** When Server state == {state,undefined,undefined,undefined,undefined,                         'source.windowslegacy.10.44.8.73',                         {shovel,                          {endpoint,                           [{amqp_params_network,<<"guest">>,<<"guest">>,                             <<"logcollection">>,"10.44.8.73",5671,0,0,0,5000,                             [{fail_if_no_peer_cert,true},                              {verify,verify_peer},                              {keyfile,                               "/etc/netwitness/ng/rabbitmq/ssl/keys/privkey.pem"},                              {certfile,                               "/etc/netwitness/ng/rabbitmq/ssl/keys/cert.pem"},                              {cacertfile,                               "/etc/netwitness/ng/rabbitmq/ssl/truststore.pem"}],                             [#Fun<amqp_uri.7.123484526>],                             [],[]}],                           [{'exchange.declare',0,<<"windowslegacy">>,                             <<"direct">>,false,true,false,false,false,[]},                            {'queue.declare',0,<<"shovel.windowslegacy">>,                             false,true,false,false,false,[]},                            {'queue.bind',0,<<"shovel.windowslegacy">>,                             <<"windowslegacy">>,<<"windowslegacy">>,false,                             []}]},                          {endpoint,                           [{amqp_params_direct,<<"logcollector">>,none,                             <<"logcollection">>,logcollector@localhost,none,                             []}],                           [{'exchange.declare',0,<<"windowslegacy">>,                             <<"direct">>,false,true,false,false,false,[]}]},                          3,on_confirm,#Fun<rabbit_shovel_sup.16.2804701>,                          #Fun<rabbit_shovel_sup.16.2804701>,

Cause

The proper procedure to change log collection method from Pull (Local Collector->Remote Collectors) to Push (Virtual Log Collector->Local Collectors->Destination Groups) or vice versa is to first delete the current method and then configure the new method.  However if you inadvertently configure both (i.e: Push and Pull) at the same time and then remove one, you will see errors on the Log Collector and VLC, resulting in logs not being collected.

The reason for this is because, when you remove a Remote Collector from the Log Collector or a Local Collector from the VLC, the corresponding peer certificate also gets deleted.  The [{fail_if_no_peer_cert,true} entry in the errors above indicate that the VLC/LC does not have/trust the certificate on the other side of the connection.
 

This can also be confirmed by performing the following steps in the NetWitness UI.

  1. In the NetWitness UI, navigate to Admin -> Services.
  2. Select the Log Collector or VLC and click on View -> Explore.
  3. Navigate to event-broker -> ssl in the directory tree in the left pane.
  4. Right-click on the ssl directory and click on Properties.
  5. Select trust from the drop down menu that appears in the lower-right pane.
  6. Enter op=list in the Parameters field and click the Send button.  The certificate of the corresponding peer will be noticeably absent.

Resolution

To resolve this issue, one of the following actions below may be taken:

If you need the Pull method, follow the steps below.

  1. In the NetWitness UI, navigate to Admin -> Services.
  2. Select the Log Collector and click on View -> Config.
  3. Click on the Remote Collectors tab.
  4. Select the corresponding VLC, click the Edit button, and then click OK.  This will force a certificate exchange between the LC and VLC.

If you need the Push method, follow the steps below.

  1. In the NetWitness UI, navigate to Admin > Services
  2. Select the Log Collector and click on View -> Config.
  3. Click on the Local Collectors tab.
  4. Select the corresponding LC destination, click the Edit button and then click OK.  This will force the certificate exchange between the LC and VLC.

Alternatively, you have the option to manually exchange the VLC/LC certificates via the Explore view, as documented in the Internal KB 000002001 


Product Details

NetWItness Product Set: NetWitness Logs & Network
NetWItness Product/Service Type: Log Collector
NetWItness  Version/Condition: 11.x, 12.x
Platform: CentOS, AlmaLinux


Approval Reviewer Queue

Technical approval queue