NetWitness 11.x Service Installation Failing on VLC after re-discovering host

Issue

Service installation failing on VLC after re-provisioning the host. Admin UI detects the new host, however it stays on “Establishing a connection do not refresh” page for about 5 minutes, and an error messages is received “Lost contact to the SA Server”

The inital issue was the expired node-cert.pem on the VLC. Running the rescue for the node-cert had failed, therefore a reporvisioning of the host was suggested. Upon the failing of the reprovisioning as per KB 000001967 checked the files under /etc/pki/nw/trust and they were missing.

Cause

1. Check for error under /var/log/rabbitmq/rabbit\@UUID.log

 2022-06-22 23:47:08.857 [error] <0.646.0>
 
2022-06-22 23:47:08.857 [error] <0.646.0> BOOT FAILED
 
2022-06-22 23:47:08.857 [error] <0.646.0> ===========
 
2022-06-22 23:47:08.857 [error] <0.646.0> Error during startup: {error,
 
2022-06-22 23:47:08.857 [info] <0.827.0> [{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.827.0>},{registered_name,[]},{error_info,{exit,,[{nw_cert_monitor,load_certificate,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,72}]},{nw_cert_monitor,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,48}]},{nw_admin_worker,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_admin_worker.erl"},{line,32}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}},{nw_admin,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.826.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.828.0>,normal}]},{links,[<0.826.0>,<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,1598},{stack_size,28},{reductions,272}], []
 
2022-06-22 23:47:08.857 [error] <0.646.0> {nw_admin,
 
2022-06-22 23:47:08.857 [error] <0.827.0> CRASH REPORT Process <0.827.0> with 0 neighbours exited with reason: ,[{nw_cert_monitor,load_certificate,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,72}]},{nw_cert_monitor,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,48}]},{nw_admin_worker,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_admin_worker.erl"},{line,...}]},...]}}},...} in application_master:init/4 line 138
 
2022-06-22 23:47:08.858 [info] <0.44.0> Application nw_admin exited with reason: ,[{nw_cert_monitor,load_certificate,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,72}]},{nw_cert_monitor,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,48}]},{nw_admin_worker,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_admin_worker.erl"},{line,...}]},...]}}},...}
 
2022-06-22 23:47:08.857 [error] <0.646.0> ,[{nw_cert_monitor,load_certificate,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,72}]},{nw_cert_monitor,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},{line,48}]},{nw_admin_worker,init,1,[{file,"/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_admin_worker.erl"},{line,...}]},...]}}},...}
 
2022-06-22 23:47:08.858 [error] <0.646.0> {failed_to_start_child,nw_admin_worker,
 
2022-06-22 23:47:08.858 [error] <0.646.0> ,
 
2022-06-22 23:47:08.858 [error] <0.646.0> [{nw_cert_monitor,load_certificate,1,
 
2022-06-22 23:47:08.858 [error] <0.646.0> [{file,
 
2022-06-22 23:47:08.859 [error] <0.646.0> "/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},
 
2022-06-22 23:47:08.859 [error] <0.646.0> {line,72}]},
 
2022-06-22 23:47:08.859 [error] <0.646.0> {nw_cert_monitor,init,1,
 
2022-06-22 23:47:08.859 [error] <0.646.0> [{file,
 
2022-06-22 23:47:08.859 [error] <0.646.0> "/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_cert_monitor.erl"},
 
2022-06-22 23:47:08.859 [error] <0.646.0> {line,48}]},
 
2022-06-22 23:47:08.859 [error] <0.646.0> {nw_admin_worker,init,1,
 
2022-06-22 23:47:08.859 [error] <0.646.0> [{file,
 
2022-06-22 23:47:08.859 [error] <0.646.0> "/home/sabuild/workspace/ngc-pipeline-linux/src/NwLogCollection/nw_admin/nw_admin_worker.erl"},
 
2022-06-22 23:47:08.860 [error] <0.646.0> {line,32}]},
 
2022-06-22 23:47:08.860 [error] <0.646.0> {gen_server,init_it,2,
 
2022-06-22 23:47:08.860 [error] <0.646.0> [{file,"gen_server.erl"},{line,417}]},
 
2022-06-22 23:47:08.860 [error] <0.646.0> {gen_server,init_it,6,
 
2022-06-22 23:47:08.860 [error] <0.646.0> [{file,"gen_server.erl"},{line,385}]},
 
2022-06-22 23:47:08.860 [error] <0.646.0> {proc_lib,init_p_do_apply,3,
 
2022-06-22 23:47:08.867 [error] <0.646.0> [{file,"proc_lib.erl"},{line,226}]}]}}},
 
2022-06-22 23:47:08.867 [error] <0.646.0> {nw_admin,start,[normal,[]]}}}}
 
2022-06-22 23:47:08.867 [error] <0.646.0>

2. chef-solo.log

 [2022-06-21T23:10:27+00:00] INFO: Processing execute[daemon-reload] action nothing (/var/lib/netwitness/config-management/cache/cookbooks/systemd/libraries/resource_factory.rb line 131)
 
[2022-06-21T23:10:27+00:00] INFO: Processing file[/etc/systemd/system/ntpd.service.d/ntpd-opts-managed.conf] action create (/var/lib/netwitness/config-management/cache/cookbooks/systemd/libraries/resource_factory.rb line 143)
 
[2022-06-21T23:10:27+00:00] INFO: Processing service[ntpd] action enable (nw-ntp::services line 35)
 
[2022-06-21T23:10:27+00:00] INFO: Processing service[ntpd] action start (nw-ntp::services line 35)
 
[2022-06-21T23:10:27+00:00] INFO: Processing template[/etc/sysctl.d/100-nw-ntp.conf] action create (nw-ntp::kparam line 9)
 
[2022-06-21T23:10:27+00:00] INFO: Processing nw_base_filesystem[nw-dns-client] action apply (nw-dns-client::filesystem line 8)
 
[2022-06-21T23:10:27+00:00] INFO: Processing ohai[reload rsa_nw_hosts plugin] action reload (nw-dns-client::hosts line 21)
 
[2022-06-21T23:10:27+00:00] INFO: Processing ruby_block[abort chef] action run (nw-dns-client::hosts line 39)
 
[2022-06-21T23:10:27+00:00] FATAL: No valid NW hosts data was available, aborting
 
[2022-06-21T23:10:27+00:00] INFO: Running queued delayed notifications before re-raising exception
 
[2022-06-21T23:10:27+00:00] ERROR: Running exception handlers
 
[2022-06-21T23:10:27+00:00] ERROR: Exception handlers complete
 
[2022-06-21T23:10:27+00:00] FATAL: Stacktrace dumped to /var/lib/netwitness/config-management/cache/chef-stacktrace.out
 
[2022-06-21T23:10:27+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

3. Rabbitmq service will fail to start.

 systemctl status rabbitmq-server -l
 
● rabbitmq-server.service - RabbitMQ broker
 
 Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
 
 Drop-In: /etc/systemd/system/rabbitmq-server.service.d
 
 └─performance-overrides.conf
 
 Active: activating (start) since Wed 2022-06-22 23:51:35 UTC; 1s ago
 
Main PID: 4899 (beam.smp)
 
 CGroup: /system.slice/rabbitmq-server.service
 
 ├─4899 /usr/lib64/erlang/erts-11.1.3/bin/beam.smp -W w -K true -A 64 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa -noshell -noinput -s rabbit boot -boot start_sasl -lager crash_log false -lager handlers []
 
 ├─5008 erl_child_setup 4096
 
 ├─5038 /usr/lib64/erlang/erts-11.1.3/bin/epmd -daemon
 
 ├─5065 inet_gethost 4
 
 └─5066 inet_gethost 4
 
Jun 22 23:51:35 VLC01 systemd[1]: Starting RabbitMQ broker...
 
Jun 22 23:51:36 VLC01 beam.smp[4899]: OWB:ERROR:UNSUPPORTED:EVP_md4
 
Jun 22 23:51:36 VLC01 beam.smp[4899]: OWB:ERROR:UNSUPPORTED:EVP_ripemd160
 
Jun 22 23:51:36 VLC01 beam.smp[4899]: OWB:ERROR:UNSUPPORTED:EVP_bf_cbc
 
Jun 22 23:51:36 VLC01 beam.smp[4899]: OWB:ERROR:UNSUPPORTED:EVP_bf_cfb64
 
Jun 22 23:51:36 VLC01 beam.smp[4899]: OWB:ERROR:UNSUPPORTED:EVP_bf_ofb
 
Jun 22 23:51:36 VLC01 beam.smp[4899]: OWB:ERROR:UNSUPPORTED:EVP_bf_ecb
 
Jun 22 23:51:36 VLC01 rabbitmq-server[4899]: Configuring logger redirection

4. Restarting the service will fail as well.

 systemctl restart rabbitmq-server
 
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
 
[root@VLC01 ~]# systemctl status rabbitmq-server -l
 
● rabbitmq-server.service - RabbitMQ broker
 
 Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
 
 Drop-In: /etc/systemd/system/rabbitmq-server.service.d
 
 └─performance-overrides.conf
 
 Active: activating (auto-restart) (Result: exit-code) since Wed 2022-06-22 23:53:10 UTC; 1s ago
 
 Process: 6091 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
 
Main PID: 6091 (code=exited, status=1/FAILURE)

Resolution

1. Take a backup of the /etc/hosts file on the VLC

2. Check for files under the directory /etc/pki/nw/trust and move the files from any working node-x to the VLC, and follow the steps below:

3. Get the node ID of the VLC

 cat /etc/salt/minion

4. On Node-Zero
a. Remove the component host from the Hosts view in the RSA NetWitness UI.
b. Remove the UUID of the component host by running the command below

 orchestration-cli-client --remove-key <UUID>

5. On Node-X
a. Run the steps for internal KB 000001967 - How to migrate an existing core appliance to a new nw-node-zero in RSA NetWitness Platform 11.x
b. Run nwsetup-tui
c. Discover the host on the UI and install the VLC service.
d. The service should be installed successfully.

Internal Comments

Refer to KB 000001967 for MovetoNodeZero script.

Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance, ESA, Endpoint Servers
RSA Version/Condition: 11.x
Platform: CentOS 7

Approval Reviewer Queue

Technical approval queue