RSA NetWitness not booting after upgrading to 11.4.x dependency failed for /boot

Issue

After NetWitness upgrade to 11.4.x and rebooting the appliance to complete the upgrade, the appliance does not come back online.

The appliance console shows that the boot has failed with error "Dependency failed for /boot" and the boot stops at maintenance mode login.

Cause

Some NetWitness Series 4S appliances were shipped using the internal SD cards as the boot partition.

NetWitness 11.4.x includes an additional security configuration which disables USB storage, like the SD card, so the new OS cannot mount the /boot partition.

Additional Notes:

1. The following NetWitness Advisory recommends to Customers to re-install NetWitness on these appliances with the SD card disabled in the BIOS, RSA Security Analytics SD Card Bulletin.
2. The NetWitness Series 4S hardware has reached End of Product Support (EOPS) after June-2019.
   For continued hardware support through to June 2021, Advanced Hardware Replacement maintenance needs to be purchased on top of the normal maintenance contract.
   
   Reference: Product Version Life Cycle for RSA NetWitness Platform

Workaround

Another temporary work-around is to boot the appliance with the old kernel.

Reboot the appliance and monitor the appliance console for the appearance of the Grub Menu.



Press the down arrow key to select the previous kernel version and then press Enter.

Resolution

Resolving the Issue when already in Maintenance Mode :
To fix this at the emergency mode prompt, login with the root password.
If the known root password does not work, then try the default password, netwitness.

In the /etc/modprobe.d/ directory look for the file, disable-usb-storage.conf, move or remove the file, and then reboot the appliance. The OS should now be able to see the SD card and boot with the new kernel version.

Prevention of this occurring in the future:
As of 11.4, the creation of this file is done as part of an upgrade or any other process that involves running Chef which means the file will return again, and then you will be in the same position again upon reboot. You have two options here: The absolute permanent fix is to reimage the device with the SD cards disabled completely; the other is to try the following:
This '/etc/modprobe.d/disable-usb-storage.conf file' is created if the manage-stig-control script group 7 is enabled for this device; this group is enabled by default. This script is ran on the Admin Server targeting any host that may have this issue. You can also go ahead and apply it to all host if all devices are at least 11.3.1.0 already.

Once the script is ran, please confirm that the above file is no longer in existence.

Note, if you are on an older version, such as 11.2, you could complete the upgrade as normal but before you do the mandated reboot, run this script to ensure that this file is not created or manually move the file as dictated in the section above.

Also note while disabling this will resolve this issue from occurring again, you may not be completely compliant with STIG any longer, if that is important to you. Please review the STIG guide for more details on what this action will do. https://community.rsa.com/docs/DOC-110202

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.4.x
Platform: CentOS
Platform (Other): NetWitness Series 4S appliance
O/S Version: 7

Summary

After NetWitness upgrade to 11.4.x the boot fails with Dependency failed for /boot and boot stops at maintenance mode login.

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue