Troubleshooting Installation and Upgrade Issues in 12.3

1. Manually modify the GRUB boot line to FIPS=0 to get it to boot.

2. From here, disable FIPS using the following command:

   manage-stig-controls --disable-control-groups 3 --host-all

3. Verify the line FIPS=1 is removed from /boot/grub2/grub.cfg

• If not, run the following command:

  grub2-mkconfig -o /boot/grub2/grub.cfg

4. Reboot.

5. Run the following command to enable FIPS:

   manage-stig-controls --enable-control-groups 3 --host-all

6. Reboot again.

Reset your deploy_admin password password.

1. On the NW Server host only, run the following command.
   nw-manage --update-deploy-admin-pw
   Please enter the new deploy_admin account password:
   Please confirm the new deploy_admin account password:
2. Review the output of the nw-manage --update-deploy-admin-pw command to verify the deploy_admin password was successfully updated on all hosts. If an NW host is down or fails for any reason as displayed by the output of the nw-manage --update-deploy-admin-pw command, run nw-manage --sync-deploy-admin-pw --host-key to synchronize the password between the NW Server and the host that failed once the communication failure is resolved.
3. On the host that failed installation or orchestration, run the nwsetup-tui command and use the new deploy_admin password in response to the Deployment Password prompt.

1. Try to update again.

2. If it fails again with the same error, try to update using the offline methods as described in "Offline Method from Hosts View" or "Offline Method Using Command Line Interface" in the Upgrade Guide for NetWitness Platform. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.

3. If you are still not able to update, contact Customer Support.

If you are upgrading from NetWitness Platform 11.x.x.x to 11.6.x.x or later, offline UI upgrade fails with the Download error message.

1. In the Command Line Interface (CLI):

   1. SSH to NW Server.

   2. Run the following command:
      upgrade-cli-client --upgrade --host-key --version
      For example:
      upgrade-cli-client --upgrade --host-key --version 11.6.0.0
2. After the NW Server is successfully updated, log in to the NW Server user interface and go to (Admin) > Hosts, where you are prompted to reboot the host.

3. Click Reboot Host from the toolbar.

   You can upgrade all the other hosts directly from the user interface:

   1. Click Begin Update from the Update Available dialog.
      After the host is upgraded, it prompts you to reboot the host.
   2. Click Reboot Host from the toolbar.

Error deploying version is displayed in the Initialize Update Package for NetWitness Platform dialog after you click on Initialize Update if the update package is corrupted.

1. Click Close to close the dialog.

2. Remove the version folder from staging folder.

3. Make sure that the salt-master service is running.

4. Recopy the update package zip file to the staging folder.

5. In the Hosts view toolbar, select Check for Updates again.
   

6. Click Initialize Update.
7. Click Update > Update Hosts from the toolbar.
8. Click Begin Update from the Update Available dialog.
   After the host is updated, it prompts you to reboot the host.
9. Click Reboot from the toolbar.

Received an error in the error log similar to the following when trying to update to version 11.6 or later:

To resolve the issue, follow the below steps.

1. SSH to Admin Server.
2. Locate the component descriptor file by running the following command.
   cd /etc/netwitness/component-descriptor/
3. Open the component descriptor file by running the following command.
   vi nw-component-descriptor. json
4. Search for “packages” section for the component you have custom build/rpm. For example, below shown is the package details for “concentrator” host that has custom build/rpm.
   “concentrator”: {
   “cookbook_name”: “rsa-concentrator”,
   “service_names”: [“rsa-nw-concentrator”],
   “family”: “launch”,
   “default_port”: xxxx, “description”: “Concentrator”,
   “packages”:[{ “name”: “rsa-nw-concentrator”,
   “version” : “11.6.0.0-2003001075220.5.cecf24b.e.17.centos”
   },
5. Delete the complete version details including (,) character in the packages section. For example, it should look like as shown below after you delete the version details.
   “packages”: [{
   “name”: “rsa-nw-concentrator”
   },

Note: You must delete the version details for all the host that has custom builds/rpms in the component descriptor of the admin server.

1. Run the upgrade process again.

Received an error similar to the following error when trying to update to a new version from the :
.Repository 'nw-rsa-base': Error parsing config: Error parsing "baseurl = 'https://nw-node-zero/nwrpmrepo / /RSA'": URL must be http, ftp, file or https not ""

Make sure that:

• the URL does exist on the NW Server host.
• you used the correct path and remove any spaces from it.

1. Try to apply the version update to the host again.
   Often this is all you need to do.
2. If you still cannot apply the new version update:
   Monitor the following logs on NW Server as it progresses (for example, run the tail -f command from the command line):
   /var/netwitness/uax/logs/sa.log
   /var/log/netwitness/orchestration-server/orchestration-server.log
   /var/log/netwitness/deployment-upgrade/chef-solo.log
   /var/log/netwitness/config-management/chef-solo.log
   /var/lib/netwitness/config-management/cache/chef-stacktrace.out
   The error appears in one or more of these logs.
3. If you still cannot apply the update, gather the logs from step 2 and contact Customer Support.

Initialize Update for Version xx.x.x.x
Missing the following update package(s)

Download Packages from NetWitness Link

1. Click Download Packages from NetWitness Community in the Initialize Update Package for NetWitness Platform dialog.
   The NetWitness Community page that contains the update files for the selected version is displayed.

2. Select the missing packages from the staging folder.
   The Initialize Update Package for NetWitness Platform dialog is displayed telling you that it is ready to initialize the update packages.

The following example illustrates an ssh error that can occur when the ssh client is run from a host with OpenSSL 1.1.x installed:
$ ssh root@10.1.2.3
ssh_dispatch_run_fatal: Connection to 10.1.2.3 port 22: message authentication code incorrect

Advanced users who want to ssh to a NetWitness Platform host from a client that is using OpenSSL 1.1.x encounter this error because of incompatibility between CENTOS 7.x and OpenSSL 1.1.x. For example:

$ rpm -q openssl
openssl-1.1.1-8.el8.x86_64

Specify the compatible cipher list on the command line. For example:

$ ssh -oCiphers=aes128-ctr,aes192-ctr,aes256-ctr root@10.1.2.3

I've read & consent to terms in IS user agreement.

root@10.1.2.3's password:

Last login: Mon Oct 21 19:03:23 2019

The /var/log/netwitness/orchestration-server/orchestration-server.log has an error similar to the following error:
API|Failure /rsa/orchestration/task/update-config-management [counter=10 reason=IllegalArgumentException::Version '11.x.x.n' is not supported

You have two options:

• Update the non-NW Server host to 11.6.0.0 or later, or
• Do not update the non-NW Server host (keep it at its current version)

You receive a message in the User Interface to reboot the host after you update and reboot the host offline.

Reboot the host in the Host View in the User Interface.

In some cases, after you upgrade to 11.6 or later from versions of 11.x, such as 11.4, the Reporting Engine service attempts to restart continuously without success.

The database files for live charts, alert status, or report status may not be loaded successfully as the files may be corrupted.

To resolve the issue, do the following:

1. Check which database files are corrupted:

   Navigate to the file located at /var/netwitness/reserver/rsa/soc/reporting-engine/logs/reporting-engine.log and check the following blocks:

   • If the live charts db file is corrupted, the following logs are displayed:

     

   • If the alert status db file is corrupted, the following logs are displayed:

     

   • If the report status db file is corrupted, the following logs are displayed:

     org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool [90030-196]

2. To resolve the live charts database file corruption, perform the following steps:

   1. Stop the Reporting Engine service.

   2. Move the livechart.mv.db file from /var/netwitness/reserver/rsa/soc/reporting-engine/livecharts folder to a temporary location.

   3. Restart the Reporting Engine service.

      Note: Some live charts data may be lost on performing the above steps.

   To resolve the alert status or report status database file corruption, perform the following steps:

   1. Stop the Reporting Engine service.
   2. Replace the corrupted db file with the latest alertstatusmanager.mv.db or reportstatusmanager.mv.dbReporting Engine restarts After upgrade to NetWitness Platform 11.4.,,,,,,, ,,,,,,, ,,,,,,, ,,,,,,, the Reporting Engine service does not restart.