Global audit logging Troubleshooting in RSA NetWitness Platform 11.x

Issue

Audit logging fails with the below errors observed in /var/log/logstash/logstash-plain.log

Logstash.log

[2019-12-20T05:23:53,633][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Authentication with RabbitMQ failed or RabbitMQ version used does not support AMQP 0-9-1. Username: logstash, vhost: /rsa/system, password length: 36. Please check your configuration.", :exception=>"MarchHare::AuthenticationFailureError"}
[2019-12-20T05:23:54,642][WARN ][com.rabbitmq.client.NullTrustManager] This trust manager trusts every certificate, effectively disabling peer verification. This is convenient for local development but prone to man-in-the-middle attacks. Please see http://www.rabbitmq.com/ssl.html#validating-cerficates to learn more about peer certificate validation.
[2019-12-20T05:23:54,691][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Authentication with RabbitMQ failed or RabbitMQ version used does not support AMQP 0-9-1. Username: logstash, vhost: /rsa/system, password length: 36. Please check your configuration.", :exception=>"MarchHare::AuthenticationFailureError"}

Resolution

Check basic troubleshooting as specified in the guide below:

https://community.rsa.com/docs/DOC-80364

If still, logstash is unable to connect to rabbitmq, follow the steps below:

 

1. Re-save the global-audit configuration:
   •  navigate to admin->system->global-auditing->try saving the audit logging configuration
   • This should automatically trigger the setconfig.sh script under /etc/logstash/setconfig.sh
   
   The script run can be verified in chef-solo.log under:
   
   /var/log/netwitness/config-management/chef-solo.log
    
2. Restart the below services:
   • logstash - service logstash restart
   • rabbitmq - service rabbitmq-server restart
   
   Audit logs (syslogs) is forwarded to log decoder for parsing once you see the logstash connected to rabbitmq as observed in the logstash-plan.log:
   
   [2019-12-27T07:29:20,495][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>32, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>4000}
   [2019-12-27T07:29:20,508][INFO ][logstash.pipeline        ] Pipeline main started
   [2019-12-27T07:29:20,551][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
   [2019-12-27T07:29:20,864][INFO ][logstash.inputs.rabbitmq ] Connected to RabbitMQ at
    
3. If you are still observing logstash and rabbitmq authentication issue follow the steps below:
   • Stop logstash service - service logstash stop
   • Delete logstash user - rabbitmqctl delete_user logstash
   •  Move /etc/netwitness/platform/logstash_rabbit/rsa-audit-server.rabbitmq.properties to other location 
     •  mv /etc/netwitness/platform/logstash_rabbit/rsa-audit-server.rabbitmq.properties /root/
   •  Run the cookbook rsa-audit-server 
     • chef-client -r "recipe[rsa-audit-server]" --config /var/lib/netwitness/config-management/client.rb --json-attributes /etc/netwitness/config-management/node.json
   
   This should resolve the logstash and rabbitmq connection issue.
   
   If you are unsure of any of the steps kindly contact RSA support.

Product Details

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue