Error message xfs_log_force: error 5 is reported on an RSA Security Analytics appliance

Issue

The /var/log/messages file reports an error similar to the following:
The appliance services are unable to start.

Resolution

In order to resolve the issue, follow the steps below.

1. Connect to the appliance via SSH as the root user.
    
2. Shutdown the nwdecoder and/or nwconcentrator services with the following command:
   stop <service>
3. Issue the df -h command.
    
4. Record the filesystem locations for the following mount points:
   • /var/netwitness/concentrator/sessiondb
   • /var/netwitness/concentrator/index
   • /var/netwitness/concentrator/metadb
   • /var/netwitness/concentrator
5. Issue the following commands:
   # umount /var/netwitness/concentrator/sessiondb
   # umount /var/netwitness/concentrator/index
   # umount /var/netwitness/concentrator/metadb
   # umount /var/netwitness/concentrator
6. Issue the df -h command again to ensure that the devices have been successfully unmounted.  The unmounted systems should no longer appear when issuing the command.
7. Navigate to the /dev/mapper directory with the following command:
   # cd /dev/mapper
8. Issue the following command:
   # ls -l
9. Run the following command against each filesystem recorded in Step 4:
   # xfs_repair <filesystem_location>
10. If xfs_repair finds an error, run the command again against the same filesystem until it no longer finds any errors.
     
11. Mount all partitions with the following command:
    # mount -a
12. Start the nwdecoder and/or nwconcentrator services with the following command:
    # start <service>

If this does not solve your issue, please open a case with RSA Technical Support and reference this article so that we may better assist you.

Internal Comments

Robert Fulton -- 28 Jul 2014
Created the article.  

Jeff Shurtliff -- 5 Aug 2014
Reviewed article as it was approved without a formal technical review. Modified statements and formatting to adhere to Primus best practices and provided additional clarification steps in the workaround.

Jeff Shurtliff -- 31 Oct 2014
Corrected some characters that didn't convert over properly from Primus.

Jeff Shurtliff -- 12 Feb 2018
Updated the article to adhere to current best practices and branding guidelines.  No technical details were modified.

Erica Chalfin - 13 Feb 2018
Perfprmed KCS review.  Changed URL name from a67104-Error-message-xfs-log-force-error-5-is-reported-on-an-RSA-Security-Analytics-appliance to match title.

Meghana Arvind -- 29 Aug 2019

• Added # in front of any commands where necessary
• Changed fonts of service names, file names, directories, and commands to Courier New
• Changed the font color of error messages to red
• Applies to:
  • RSA Product Set: changed "NetWitness Logs & Packets" to "NetWitness Logs & Network"
• Resolution: Changed the following
  • from: "If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and quote this article number for further assistance."
  • to: "If this does not solve your issue, please open a case with RSA Technical Support and reference this article so that we may better assist you. 

Product Details

RSA Product Set: Security Analytics, NetWitness Logs & Network
RSA Product/Service Type: Decoder, Log Decoder, Concentrator, Hybrid, Broker
RSA Version/Condition: 10.3.x, 10.4.x, 10.5.x, 10.6.x
Platform: CentOS
O/S Version: 6

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue