.png?width=180&height=32&name=NW%20Logo%20(2).png){kind=small}
Global Audit Logging Stops after Rabbitmq Service Restart on the Admin Server in RSA NetWitness Platform
Issue
Global Auditing could be configured on RSA NetWitness Platform as per - NW Cfg: Configure Global Audit LoggingHowever, when rabbitmq-server service is restarted on the Admin Server/Node Zero the Global Audit Logging stops and customers are unable to see the Audit logs in Investigation.
You may see below Error and Warning messages related to Logstash Service on the Admin Server -
[root@NWSERVER ~]# systemctl status logstash
Dec 13 18:22:48 NWSERVER logstash[1121]: Caught exception when recovering queue ls-audit
Dec 13 18:22:48 NWSERVER logstash[1121]: Caught exception when recovering consumer amq.xxxxx.xxxx.xxxxx
[root@NWSERVER ~]# tail -100 /var/log/logstash/logstash.log
{:timestamp=>"2018-04-26T11:50:07.386000+0000", :message=>"RabbitMQ connection error: Connection to localhost:5672 refused. Will reconnect in 10 seconds...", :level=>:error}
{:timestamp=>"2018-04-26T11:50:17.569000+0000", :message=>"RabbitMQ connection error: NOT_FOUND - no exchange 'carlos.audit' in vhost '/rsa/system'. Will reconnect in 10 seconds...", :level=>:error}
{:timestamp=>"2018-04-26T11:54:35.711000+0000", :message=>"RabbitMQ connection error: NOT_FOUND - no exchange 'carlos.audit' in vhost '/rsa/system'. Will reconnect in 10 seconds...", :level=>:error}
[root@NWSERVER ~]# tail -1000 /var/log/logstash/logstash-plain.log
[2019-12-13T18:22:48,580][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqps://logstash:XXXXXX@32c5b77d-309d-45ea-9134-9cd5c04791d8:5671/rsa/system", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)}
Dec 13 18:22:48 NWSERVER logstash[1121]: Caught exception when recovering queue ls-audit
Dec 13 18:22:48 NWSERVER logstash[1121]: Caught exception when recovering consumer amq.xxxxx.xxxx.xxxxx
[root@NWSERVER ~]# tail -100 /var/log/logstash/logstash.log
{:timestamp=>"2018-04-26T11:50:07.386000+0000", :message=>"RabbitMQ connection error: Connection to localhost:5672 refused. Will reconnect in 10 seconds...", :level=>:error}
{:timestamp=>"2018-04-26T11:50:17.569000+0000", :message=>"RabbitMQ connection error: NOT_FOUND - no exchange 'carlos.audit' in vhost '/rsa/system'. Will reconnect in 10 seconds...", :level=>:error}
{:timestamp=>"2018-04-26T11:54:35.711000+0000", :message=>"RabbitMQ connection error: NOT_FOUND - no exchange 'carlos.audit' in vhost '/rsa/system'. Will reconnect in 10 seconds...", :level=>:error}
[root@NWSERVER ~]# tail -1000 /var/log/logstash/logstash-plain.log
[2019-12-13T18:22:48,580][WARN ][logstash.inputs.rabbitmq ] RabbitMQ connection was closed! {:url=>"amqps://logstash:XXXXXX@32c5b77d-309d-45ea-9134-9cd5c04791d8:5671/rsa/system", :automatic_recovery=>true, :cause=>com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)}
Customers with large environments may also see high utilization of disk space in /var/netwitness partition of the Admin Server and slowness in the Netwitness UI.
ls.audit queue in Rabbitmq on the Admin Server will be growing in messages and will not be consuming.
[root@NWSERVER ~]# rabbitmqctl list_queues -p /rsa/system | grep ls-audit
ls-audit 98491
ls-audit 98491
Cause
Global Audit Logging is mainly facilitated by the Logstash service and the Rabbitmq service acts as the reliable Message Bus.However, when the Rabbitmq Service is restarted or crashed logstash service does not recover the connection to the relevant vhost (/rsa/system) and queue (ls.audit) in Rabbitmq. Therefore logstash service should also be manually restarted to reconnect to the RabbitMQ.
Workaround
The workaround would be to override the systemd configuration of logstash to get restarted upon RabbitMQ service restart."PartOf" systemd option could be used to restart logstash service as a part of rabbitmq-server service restart.
Add the option "PartOf=rabbitmq-server.service" under the [Unit] section in the logstash.service systemd file.
[root@NWSERVER ~]# vi /etc/systemd/system/logstash.service
[Unit]
Description=logstash
PartOf=rabbitmq-server.service
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
ExecStartPre=/etc/systemd/system/logstash-prestart.sh
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target
[Unit]
Description=logstash
PartOf=rabbitmq-server.service
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
ExecStartPre=/etc/systemd/system/logstash-prestart.sh
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target
After that restart the rabbitmq-server service.
# systemctl daemon-reload
# systemctl restart rabbitmq-server.service
# systemctl restart rabbitmq-server.service
Resolution
Currently this is a known issue and will be fixed in RSA Netwitness 11.5 Version.
Notes
Log files to check are -/var/log/logstash/logstash.log
/var/log/logstash/logstash-plain.log
/var/netwitness/logstash/logs/rsa-netwitness-audit.log
Internal Comments
Suneth Jayarathne -- 13-Dec-2019Created the Article
Product Details
RSA Product Set: RSA NetWitness PlatformRSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.2.x, 11.3.x, 11.4.x
Summary
Global Auditing stops working upon Rabbitmq Server Restart
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue