RSA Security Analytics - Unable to SSH to appliance due to hung_task_timeout

Issue

Appliance is unresponsive when SSH-ing to. After connecting through iDRAC the below error is shown.

“echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.” and “blocked for more than 120 seconds” somewhere.

Cause

By default Linux uses up to 40% of the available memory for file system caching. After this mark has been reached the file system flushes all outstanding data to disk causing all following I/Os going synchronous.

There is a time limit of 120 seconds by default for flushing out this data to disk. In this case the I/O subsystem is not fast enough to flush the data within 120 seconds.

Resolution

Please follow below steps to fix the issue permanently.

Reboot the Appliance to get access via SSH
Edit /etc/sysctl.conf
Add below lines at the bottom of the file:

 vm.dirty_background_ratio = 5 
 
vm.dirty_ratio = 10

4. Save and Exit
5. Reboot the system

Product Details

RSA Product Set: Security Analytics
RSA Version/Condition: 10.X
Platform: CentOS
O/S Version: 6

Summary

Unable to ssh to appliance due to kernel reached hung mode. This issue can be resolved after changing parameters in /etc/sysctl.conf file.

Approval Reviewer Queue

ASOC Approval Group

RSA Security Analytics - Unable to SSH to appliance due to hung_task_timeout_secs