Rabbitmq-server service crash after modify the permission of /etc/hosts file.
Issue
User cannot access to NetWitness UI because of rabbitmq-server service crash.Following errors occur.
/var/log/rabbitmq/rabbit@
2021-08-30 10:46:23.391 [error] <0.507.0> CRASH REPORT Process <0.507.0> with 0 neighbours exited with reason: {upstream_channel_down,shutdown} in gen_server2:terminate/3 line 1183
2021-08-30 10:46:23.391 [error] <0.869.0> Supervisor {<0.869.0>,rabbit_federation_link_sup} had child {upstream,[<<"luhCr5iNcFFyeJcvxl+ibaBhPPrWCIJb+PWeYLJUdOHPxR5eWb9rGvSgr+4ibmkYnxUeXrcw8KFe90mHN2qWZQaLWitTXoozrML6FVN4DHiOX9Zit19AOhMPnEk2fblvo81s7daaxiOijXXir8C3Li6VJc7AFdluQKqTbZMuA87+KasvVjW+p0iRNS7w8i7z0+oALaXic3qXI+liSk9skJ4xenedTemk7D19q/M73FZ1NonUt5Re3LJu/D8mdyjA">>],
<<"carlos.audit">>,<<"carlos.audit">>,
<<"federation-link-carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,
1000,1,5,3600000,none,false,'on-confirm',none,
<<"carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,false} started with rabbit_federation_exchange_link:start_link({{upstream,[<<"luhCr5iNcFFyeJcvxl+ibaBhPPrWCIJb+PWeYLJUdOHPxR5eWb9rGvSgr+4ibmkYnxUeXrcw8KFe90mHN...">>],...},...}) at <0.870.0> exit with reason {upstream_channel_down,shutdown} in context child_terminated
2021-08-30 10:46:23.391 [error] <0.1805.0> ** Generic server <0.1805.0> terminating
** Last message in was {'DOWN',#Ref<0.2884530769.523763713.194220>,process,<0.1841.0>,shutdown}
** When Server state == {state,{upstream,[<<"c/aTTioTpFq1lpB24iQ1HIxRKnJH0fjXSo+Qv4SQYKMtdBaQz/VO5auOBvphFtTnWQi3wHS0lKQjHLnf4fPlYu/2kbPmvInqsuHMcK0EaDSQdr2fKZ726XL8gLx1upx7pE5JSCNAC7YbhAO6Q+XBVdZ5lqm5XpUovs4AZsz7VI9GEH/9eZkRQfqrrNfrs0Y6ZGj59xP2dhlgV+TO6Cxjs1RGSQoUA4utYRSse/LNSCymC2yL6WoGMkHA7rVNMP3H">>],<<"carlos.esm.eventsource.changes">>,<<"carlos.esm.eventsource.changes">>,<<"federation-link-carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,1000,1,5,3600000,none,false,'on-confirm',none,<<"carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,false},{upstream_params,<<"vX46I4cmWJX4eGn/ef+pCVc3786GS+iOwrb4KHQ3DBNSFctDsYToOMbwqrl/nlTIyhhkn26We1g5wd+WkEEyaOE32QVyN417hh2Lilex8TUHSslqNIavt9tDgyQVTOALWRSENGYecAF7RLJNqWQhHhANjkFytjNto7TaHxTacuaWiJ4nGfEL7kpHQh/liJVt/yzcH2HFnMLoS2IHkUHoeE4BJTCuBCShjEN20P5WL8AL/3SP2yEEtFdWU9FVbkzR">>,{amqp_params_network,<<"guest">>,<<"JvjOPNiM+pxdHzBWwLMJR3KNo5JP6g8cr9q6oESLfuxdvjPaqQgqSV+mx8eZTQSl">>,<<"/rsa/system">>,"3999d012-d370-49ee-aa19-eb86dc70af0f",5671,2047,0,10,60000,[{server_name_indication,"3999d012-d370-49ee-aa19-eb86dc70af0f"},{verify,verify_peer}],[#Fun<amqp_uri.12.131604370>],[],[]},{exchange,{resource,<<"/rsa/system">>,exchange,<<"carlos.esm.eventsource.changes">>},headers,false,false,false,[],[{federation,[]}],[{vhost,<<"/rsa/system">>},{name,<<"carlos-federate">>},{pattern,<<"^carlos\\.*">>},{'apply-to',<<"exchanges">>},{definition,[{<<"federation-upstream-set">>,<<"all">>}]},{priority,0}],undefined,{[],[rabbit_federation_exchange]},#{user => <<"<UUID>">>}},<<"amqps://3999d012-d370-49ee-aa19-eb86dc70af0f:5671">>,[{<<"uri">>,longstr,<<"amqps://3999d012-d370-49ee-aa19-eb86dc70af0f:5671">>},{<<"exchange">>,longstr,<<"carlos.esm.eventsource.changes">>}]},<<"rabbit@SELDC-HYBRID">>,<0.1830.0>,<0.1841.0>,<0.1845.0>,<<"amq.ctag-NJfroFfAO5CqDp_rWPnXIQ">>,<<"federation: carlos.esm.eventsource.changes -> rabbi...">>,...}
** Reason for termination ==
** {upstream_channel_down,shutdown}
2021-08-30 10:46:23.391 [error] <0.869.0> Supervisor {<0.869.0>,rabbit_federation_link_sup} had child {upstream,[<<"luhCr5iNcFFyeJcvxl+ibaBhPPrWCIJb+PWeYLJUdOHPxR5eWb9rGvSgr+4ibmkYnxUeXrcw8KFe90mHN2qWZQaLWitTXoozrML6FVN4DHiOX9Zit19AOhMPnEk2fblvo81s7daaxiOijXXir8C3Li6VJc7AFdluQKqTbZMuA87+KasvVjW+p0iRNS7w8i7z0+oALaXic3qXI+liSk9skJ4xenedTemk7D19q/M73FZ1NonUt5Re3LJu/D8mdyjA">>],
<<"carlos.audit">>,<<"carlos.audit">>,
<<"federation-link-carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,
1000,1,5,3600000,none,false,'on-confirm',none,
<<"carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,false} started with rabbit_federation_exchange_link:start_link({{upstream,[<<"luhCr5iNcFFyeJcvxl+ibaBhPPrWCIJb+PWeYLJUdOHPxR5eWb9rGvSgr+4ibmkYnxUeXrcw8KFe90mHN...">>],...},...}) at <0.870.0> exit with reason {upstream_channel_down,shutdown} in context child_terminated
2021-08-30 10:46:23.391 [error] <0.1805.0> ** Generic server <0.1805.0> terminating
** Last message in was {'DOWN',#Ref<0.2884530769.523763713.194220>,process,<0.1841.0>,shutdown}
** When Server state == {state,{upstream,[<<"c/aTTioTpFq1lpB24iQ1HIxRKnJH0fjXSo+Qv4SQYKMtdBaQz/VO5auOBvphFtTnWQi3wHS0lKQjHLnf4fPlYu/2kbPmvInqsuHMcK0EaDSQdr2fKZ726XL8gLx1upx7pE5JSCNAC7YbhAO6Q+XBVdZ5lqm5XpUovs4AZsz7VI9GEH/9eZkRQfqrrNfrs0Y6ZGj59xP2dhlgV+TO6Cxjs1RGSQoUA4utYRSse/LNSCymC2yL6WoGMkHA7rVNMP3H">>],<<"carlos.esm.eventsource.changes">>,<<"carlos.esm.eventsource.changes">>,<<"federation-link-carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,1000,1,5,3600000,none,false,'on-confirm',none,<<"carlos-upstream-3999d012-d370-49ee-aa19-eb86dc70af0f">>,false},{upstream_params,<<"vX46I4cmWJX4eGn/ef+pCVc3786GS+iOwrb4KHQ3DBNSFctDsYToOMbwqrl/nlTIyhhkn26We1g5wd+WkEEyaOE32QVyN417hh2Lilex8TUHSslqNIavt9tDgyQVTOALWRSENGYecAF7RLJNqWQhHhANjkFytjNto7TaHxTacuaWiJ4nGfEL7kpHQh/liJVt/yzcH2HFnMLoS2IHkUHoeE4BJTCuBCShjEN20P5WL8AL/3SP2yEEtFdWU9FVbkzR">>,{amqp_params_network,<<"guest">>,<<"JvjOPNiM+pxdHzBWwLMJR3KNo5JP6g8cr9q6oESLfuxdvjPaqQgqSV+mx8eZTQSl">>,<<"/rsa/system">>,"3999d012-d370-49ee-aa19-eb86dc70af0f",5671,2047,0,10,60000,[{server_name_indication,"3999d012-d370-49ee-aa19-eb86dc70af0f"},{verify,verify_peer}],[#Fun<amqp_uri.12.131604370>],[],[]},{exchange,{resource,<<"/rsa/system">>,exchange,<<"carlos.esm.eventsource.changes">>},headers,false,false,false,[],[{federation,[]}],[{vhost,<<"/rsa/system">>},{name,<<"carlos-federate">>},{pattern,<<"^carlos\\.*">>},{'apply-to',<<"exchanges">>},{definition,[{<<"federation-upstream-set">>,<<"all">>}]},{priority,0}],undefined,{[],[rabbit_federation_exchange]},#{user => <<"<UUID>">>}},<<"amqps://3999d012-d370-49ee-aa19-eb86dc70af0f:5671">>,[{<<"uri">>,longstr,<<"amqps://3999d012-d370-49ee-aa19-eb86dc70af0f:5671">>},{<<"exchange">>,longstr,<<"carlos.esm.eventsource.changes">>}]},<<"rabbit@SELDC-HYBRID">>,<0.1830.0>,<0.1841.0>,<0.1845.0>,<<"amq.ctag-NJfroFfAO5CqDp_rWPnXIQ">>,<<"federation: carlos.esm.eventsource.changes -> rabbi...">>,...}
** Reason for termination ==
** {upstream_channel_down,shutdown}
/var/log/rabbitmq/log/crash.log
2021-08-30 10:46:23 =CRASH REPORT====
crasher:
initial call: rabbit_federation_exchange_link:init/1
pid: <0.870.0>
registered_name: []
exception exit: {{upstream_channel_down,shutdown},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1183}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
ancestors: [<0.869.0>,<0.439.0>,rabbit_federation_exchange_link_sup_sup,rabbit_federation_sup,rabbit_sup,<0.346.0>]
message_queue_len: 0
messages: []
links: [<0.869.0>]
dictionary: [{gen_server_call_timeout,60000},{rand_seed,{#{bits => 58,jump => #Fun<rand.8.10897371>,next => #Fun<rand.5.10897371>,type => exrop,uniform => #Fun<rand.6.10897371>,uniform_n => #Fun<rand.7.10897371>,weak_low_bits => 1},[266498609438196529|203684740268443492]}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 13827527687
neighbours:
crasher:
initial call: rabbit_federation_exchange_link:init/1
pid: <0.870.0>
registered_name: []
exception exit: {{upstream_channel_down,shutdown},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1183}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
ancestors: [<0.869.0>,<0.439.0>,rabbit_federation_exchange_link_sup_sup,rabbit_federation_sup,rabbit_sup,<0.346.0>]
message_queue_len: 0
messages: []
links: [<0.869.0>]
dictionary: [{gen_server_call_timeout,60000},{rand_seed,{#{bits => 58,jump => #Fun<rand.8.10897371>,next => #Fun<rand.5.10897371>,type => exrop,uniform => #Fun<rand.6.10897371>,uniform_n => #Fun<rand.7.10897371>,weak_low_bits => 1},[266498609438196529|203684740268443492]}}]
trap_exit: true
status: running
heap_size: 987
stack_size: 27
reductions: 13827527687
neighbours:
'rabbitmqctl' command failed with following error.
# rabbitmqctl list_parameters -p /rsa/system
Error: unable to perform an operation on node 'rabbit@<UUID>'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit@<UUID>
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: ['rabbit@<UUID>']
rabbit@<UUID>:
* unable to connect to epmd (port 4369) on <UUID>: nxdomain (non-existing domain)
Current node details:
* node name: 'rabbitmqcli-126699-rabbit@<UUID>'
* effective user's home directory: /var/lib/rabbitmq
* Erlang cookie hash: /ZV/MiIec4u4nZYU6o5P6Q==
Error: unable to perform an operation on node 'rabbit@<UUID>'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit@<UUID>
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: ['rabbit@<UUID>']
rabbit@<UUID>:
* unable to connect to epmd (port 4369) on <UUID>: nxdomain (non-existing domain)
Current node details:
* node name: 'rabbitmqcli-126699-rabbit@<UUID>'
* effective user's home directory: /var/lib/rabbitmq
* Erlang cookie hash: /ZV/MiIec4u4nZYU6o5P6Q==
Cause
NetWitness admin host cannot resolve node-zero's UUID because the permission of /etc/hosts file is not 644.Resolution
- Check /etc/hosts has correct UUID of node-zero. Add the UUID if /etc/hosts does not have it.
# cat /etc/hosts
127.0.0.1 SELDC-SAHEAD localhost localhost.localdomain localhost4 localhost4.localdomain4 nw-node-zero 6d651070-22cc-466f-xxxx-xxxxxxxxxxxx 6d651070-22cc-466f-xxxx-xxxxxxxxxxxx.netwitness nw-node-zero.netwitness
::1 SELDC-SAHEAD localhost localhost.localdomain localhost6 localhost6.localdomain6 - Check the permission of /etc/hosts is 644. Change the permission if it's not 644.
# chmod 644 /etc/hosts
# ll /etc/hosts
-rw-r--r--. 1 root root 2134 Jun 29 02:20 /etc/hosts
Product Details
RSA Product Set: RSA NetWitness PlatformRSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.4, 11.5, 11.6
Platform: CentOS
O/S Version: 7
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue