Skip to content
  • There are no suggestions because the search field is empty.

Admin Server Launch Service does not trust any certificates for the RSA NetWitness Platform

Issue

Upon upgrade or an installation, it is possible that a launch service on the Admin Server may not come online. Depending on when this problem occurs and which service it occurs with, different results can be expected. The service may seem offline in the UI; some pages may become inaccessible; these are all sample possibilities.

When inspecting the logs for the relevant service (for instance, license server logs are in /var/log/netwitness/license-server/license-server.log) after a while of it being left alone, you may see messages similar to the following indicating a corrupt truststore. This could also mean that the service failed to renew it's certificate whenever a cert-renewal was called but let's assume for the purpose of this article that this is not the case. Suppose you see messages like the following:
 
2020-01-24 03:01:37,449 [main] ERROR Security|Keystore file /etc/netwitness/license-server/keystore.p12 cannot be read (java.io.IOException: Could not decrypt data.)
Or perhaps:
2020-01-24 03:01:40,736 [main] WARN Security|Certificate for CN=2946d7f0-8357-4749-bbd6-4f48497b76b1,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US issued by CN=NetWitness Root CA,
OU=NetWitness Platform, O=RSA, L=Reston, ST=VA, C=US is not trusted
The above is indicative that this service does not trust the connections being made to it and does not trust certificates that are signed by the NetWitness Root CA available in this environment. If you restart a service, the service may even hang with messages like the following but write nothing more until after some time, the not trusted messages appear again.
2020-01-29 16:27:09,617 [ main] INFO Bootstrap|Service logs will be written to /var/log/netwitness/license-server
2020-01-29 16:27:09,623 [ main] INFO Bootstrap|Service configuration will be read from /etc/netwitness/license-server
2020-01-29 16:27:09,698 [ main] INFO Bootstrap|Starting license-server.743b3a78-20b7-4072-b481-c5122a46b83d (v11.3.0.0)
2020-01-29 16:27:10,034 [ main] INFO Bootstrap|Initialized service cryptography with 4 providers (BSAFE=CRYPTOJ 6.2.2 20161215 0745, FIPS-140=true).



Cause

The circumstances that are mentioned indicate a corrupt or incomplete truststore file that is used by the launch service that needs to be reset. The below steps can be used to regenerate and orchestrate this service to a healthy state. Note that you should not do this for just any launch service.  Consult the following list before proceeding.

Resolution

The below steps can be used to complete this process but there are some services that you should never do this on unless told by CE.

The services that these steps should NOT be used on include:
  • Admin Server
  • Orchestration Server
  • Security Server
  • Config Server
If you have a similar situation for a service on this list, please contact CE before proceeding.
The steps that are mentioned below are transferable; this means that while the commands are specific to a service, the file structure and flow will be similar. The example that is provided is for license server.
  1. SSH to the Admin Server
  2. Stop the License Server.
    systemctl stop rsa-nw-license-server
  3. Create a folder that we put our backed-up files.
    mkdir /root/license-server-backup
  4. Move the systemd configuration out.
    mv /etc/systemd/system/rsa-nw-license-server.service.d/rsa-nw-license-server-opts-managed.conf /root/license-server-backup
  5. Move the Keystore and lockbox of this service to the backup folder.
    cd /etc/netwitness/license-server/
    mv keystore.p12 lockbox.ss lockbox.ss.lock /root/license-server-backup
  6. Move the Certificate files for the service to the backup folder.
    mv /etc/pki/nw/service/rsa-nw-license-server* /root/license-server-backup
  7. Move the bootstrap file marker to the backup folder.
    mv /etc/pki/nw/service/bootstrap/license-server.completed /root/license-server-bkp
  8. Run the relevant Chef Recipe for this service. Note, the naming scheme is not 1-1. Example: investigate-server's recipe is rsa-investigate. For the license server, it is called "rsa-license-server"
    chef-client -r "recipe[rsa-license-server]" --config /var/lib/netwitness/config-management/client.rb --json-attributes /etc/netwitness/config-management/node.json
After running the above steps, check and see if the logs are being written and the service is staying up and logging. One other method to check that the status is to run something like the following. All the launch services rely on the RabbitMQ message bus for communication, so a good metric to check that the health of a service is to see if it created its queue. You may see something similar to the following if the service is now healthy.
[root@NWAdmin ~]# rabbitmqctl list_queues -p /rsa/system | grep license
license-server.593e4744-81ef-4aea-b0ed-473686375e4d 0
license-server.any 0


Product Details

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Admin Server
RSA Version/Condition: 11.X
Platform: CentOS 7

Summary

This document talks about how you can reorchestrate some of the Launch devices if they have corrupt trust stores. Note, this process only works on certain launch services and should not be ran against certain ones due to their effect on the system.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue