Skip to content
  • There are no suggestions because the search field is empty.

RSA NetWitness server rabbitmq runs out of file descriptors

Issue

If you note below errors that are logged in rabbitmq-server.log, this is an indication that rabbitmq is running out of file descriptors.
2020-02-21 18:23:13.204 [warning] <0.600.0> Ranch acceptor reducing accept rate: out of file descriptors
2020-02-21 18:23:13.556 [error] <0.653.0> ** Generic server rabbit_mgmt_external_stats terminating
** Last message in was emit_update
** When Server state == {state,4096,[},undefined,5000}
** Reason for termination ==
** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd"},[binary,stderr_to_stdout,stream,in,hide,out]],[]},{os,cmd,2,[{file,"os.erl"},{line,275}]},{rabbit_mgmt_external_stats,get_used_fd,1,[{file,"src/rabbit_mgmt_external_stats.erl"},{line,97}]},{rabbit_mgmt_external_stats,get_used_fd,0,[{file,"src/rabbit_mgmt_external_stats.erl"},{line,65}]},{rabbit_mgmt_external_stats,'-infos/2-lc$^0/1-0-',2,[{file,"src/rabbit_mgmt_external_stats.erl"},{line,181}]},{rabbit_mgmt_external_stats,emit_update,1,[{file,"src/rabbit_mgmt_external_stats.erl"},{line,370}]},{rabbit_mgmt_external_stats,handle_info,2,[{file,"src/rabbit_mgmt_external_stats.erl"},{line,357}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]}]}
2020-02-21 18:23:13.556 [error] <0.653.0> CRASH REPORT Process rabbit_mgmt_external_stats with 0 neighbours crashed with reason: maximum number of file descriptors exhausted, check ulimit -n







Cause

By default, RSA NetWitness rabbitmq-server comes with limit of 3996 file descriptors, and CentOS has 4096 on files open.
It is possible due to communication issue, rabbitmq-server on each host keeps trying to connect to admin server rabbtimq server and connections files are not closed properly, so open files number grows to a point to reach maximum limit of open files on CentOS. As a result, rabbitmq-server will not accept any connection and fails.


Workaround


We can restart rabbitmq-server service as temporary workaround.

Resolution

We can increase file descriptors maximum limit for rabbitmq-server service.
  1. Get the current file_descriptors setting.
    # rabbitmqctl report |grep 'file_descriptor \| total_limit'
    {file_descriptors,
    [{total_limit,3996},
  2. Set max file open maximum limit on CentOS (Ex. 100000)
    # vi /etc/sysctl.conf
    Add this line
    fs.file-max=100000
    Save the file.
     
  3. Set file open maximum limit for each user/process (Ex. 65535). Note, this number must be smaller than fs.file-max.
    # vi /etc/security/limits.conf
    Add these lines
    * soft nproc 65535
    * hard nproc 65535
    * soft nofile 65535
    * hard nofile 65535
    Save the file.
     
  4. Load sysctl setting.
    # sysctl -p
     
  5. Change LimitNOFILE for rabbitmq-server.
    # vi /etc/systemd/system/rabbitmq-server.service.d/performance-overrides.conf
    Change LimitNOFILE=4096 to a higher value, ex LimitNOFILE=8192
    Save file
     
  6. Reload daemon
    # systemctl daemon-reload
     
  7. Restart rabbitmq-server service
    # systemctl restart rabbitmq-server
     
  8. Verify new setting is applied
    # rabbitmqctl report |grep 'file_descriptor \| total_limit'
    {file_descriptors,
    [ {total_limit,8092},

Product Details

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Rabbitmq, CentOS
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: EL7

Summary

This KB discuss about how to increase file descriptors limitation for rabbitmq server service on NetWitness 11.x.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue