Constant MessageBrokerLogReceiver errors on Appliance in RSA Security Analytics 10.5+
Issue
Constantly seeing the error message below in /var/log/messages:
[MessageBrokerLogReceiver] [failure] error 2016-09-15T06.32.11Z Error on AMQP connection
<0.16034.11>:{ssl_upgrade_error,{tls_alert,"unknown ca"}}
<0.16034.11>:{ssl_upgrade_error,{tls_alert,"unknown ca"}}
Cause
Few Factors can cause the invalidity of the CA certificate:1- Invalid CA certificates between Local LogCollector and LogDecoder
2- Invalid CA certificates between SA server and Core Appliance
Resolution
The Invalid CA certificates can be resolved by re-provisioning the appliance , which will also ensure the correct exchange of CA certs between SA server and the core services.
However, in rare circumstances, when the Hosts are removed from the UI, few RabbitMQ components are not removed , hence causing the above error.
Attemp1:
Refer to KB 000030165 for resetting puppet certificates and reprovisioning the Host
Attemp1:
If attempt 1 did not work and the above error still present, then we will need to verify the redundant component in RabbitMQ that is still causing the above error. Here is how to do :
1- ssh to SA server
2- run the command and note the results returned:
a- rabbitmqctl report | grep -i <IP ADDRESS of the HOST that is producing the error>
Example:
-rabbitmqctl report | grep -i 192.168.223.11
- Then we get the output below :
"federation-upstream carlos-upstream-um-lhybrid {"uri":"amqps://192.168.223.11?cacertfile=/etc/rabbitmq/ssl/truststore.pem&certfile=
/etc/rabbitmq/ssl/server/cert.pem&keyfile=/etc/rabbitmq/ssl/server/key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external","expires":3600000}
federation-upstream carlos-upstream-um-lhybrid {"uri":"amqps://192.168.223.11?cacertfile=/etc/rabbitmq/ssl/truststore.pem&certfile=
/etc/rabbitmq/ssl/server/cert.pem&keyfile=/etc/rabbitmq/ssl/server/key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external","expires":3600000}"
/etc/rabbitmq/ssl/server/cert.pem&keyfile=/etc/rabbitmq/ssl/server/key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external","expires":3600000}
federation-upstream carlos-upstream-um-lhybrid {"uri":"amqps://192.168.223.11?cacertfile=/etc/rabbitmq/ssl/truststore.pem&certfile=
/etc/rabbitmq/ssl/server/cert.pem&keyfile=/etc/rabbitmq/ssl/server/key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external","expires":3600000}"
b- The above redundant entries in RabbitMQ are the root cause of the error seen above
c- run the command below to removed them:
a- rabbitmqctl -q clear_parameter -p /rsa/sa federation-upstream carlos-upstream-<HOSTNAME>
b- rabbitmqctl -q clear_parameter -p /rsa/system federation-upstream carlos-upstream-<HOSTNAME>
Example:
rabbitmqctl -q clear_parameter -p /rsa/sa federation-upstream carlos-upstream-um-lhybrid
rabbitmqctl -q clear_parameter -p /rsa/system federation-upstream carlos-upstream-um-lhybrid
rabbitmqctl -q clear_parameter -p /rsa/system federation-upstream carlos-upstream-um-lhybrid
The CA certificate error should have now been cleared
Product Details
RSA Product Set: Security AnalyticsRSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.5+
Approval Reviewer Queue
ASOC Approval Group