Reporting engine service stopped alarm appears every day at a certain time
Issue
Customer is receiving "Reporting engine service stopped" alarm triggered every day at a certain time.You are able to see following logs repeated every day from /var/log/messages in admin server.
Jul 17 00:01:56 HOSTNAME ReportingEngine_Backup: Starting backup script for reporting-engine version 11.6.1.0 .
Jul 17 00:01:56 HOSTNAME ReportingEngine_Backup: Creating backup directory (if they don't exist).
Jul 17 00:01:56 HOSTNAME ReportingEngine_Backup: Stopping reporting-engine Service
Jul 17 00:01:56 HOSTNAME systemd: Stopping Reporting Engine...
Jul 17 00:01:56 HOSTNAME reporting-engine.sh: Sun Jul 17 00:01:56 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh stop
Jul 17 01:11:10 HOSTNAME ReportingEngine_Backup: Successfully created backup for /var/netwitness/re-server
Jul 17 01:11:10 HOSTNAME ReportingEngine_Backup: Starting Reporting Engine Service
Jul 17 01:11:10 HOSTNAME systemd: Started Reporting Engine.
Jul 17 01:11:10 HOSTNAME reporting-engine.sh: Sun Jul 17 01:11:10 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh restart_blocking_mode
.
.
Jul 18 00:02:00 HOSTNAME ReportingEngine_Backup: Starting backup script for reporting-engine version 11.6.1.0 .
Jul 18 00:02:00 HOSTNAME ReportingEngine_Backup: Creating backup directory (if they don't exist).
Jul 18 00:02:00 HOSTNAME ReportingEngine_Backup: Stopping reporting-engine Service
Jul 18 00:02:00 HOSTNAME systemd: Stopping Reporting Engine...
Jul 18 00:02:00 HOSTNAME reporting-engine.sh: Mon Jul 18 00:02:00 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh stop
Jul 18 01:10:12 HOSTNAME ReportingEngine_Backup: Successfully created backup for /var/netwitness/re-server
Jul 18 01:10:12 HOSTNAME ReportingEngine_Backup: Starting Reporting Engine Service
Jul 18 01:10:12 HOSTNAME systemd: Started Reporting Engine.
Jul 18 01:10:12 HOSTNAME reporting-engine.sh: Mon Jul 18 01:10:12 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh restart_blocking_mode
.
.
Jul 17 00:01:56 HOSTNAME ReportingEngine_Backup: Creating backup directory (if they don't exist).
Jul 17 00:01:56 HOSTNAME ReportingEngine_Backup: Stopping reporting-engine Service
Jul 17 00:01:56 HOSTNAME systemd: Stopping Reporting Engine...
Jul 17 00:01:56 HOSTNAME reporting-engine.sh: Sun Jul 17 00:01:56 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh stop
Jul 17 01:11:10 HOSTNAME ReportingEngine_Backup: Successfully created backup for /var/netwitness/re-server
Jul 17 01:11:10 HOSTNAME ReportingEngine_Backup: Starting Reporting Engine Service
Jul 17 01:11:10 HOSTNAME systemd: Started Reporting Engine.
Jul 17 01:11:10 HOSTNAME reporting-engine.sh: Sun Jul 17 01:11:10 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh restart_blocking_mode
.
.
Jul 18 00:02:00 HOSTNAME ReportingEngine_Backup: Starting backup script for reporting-engine version 11.6.1.0 .
Jul 18 00:02:00 HOSTNAME ReportingEngine_Backup: Creating backup directory (if they don't exist).
Jul 18 00:02:00 HOSTNAME ReportingEngine_Backup: Stopping reporting-engine Service
Jul 18 00:02:00 HOSTNAME systemd: Stopping Reporting Engine...
Jul 18 00:02:00 HOSTNAME reporting-engine.sh: Mon Jul 18 00:02:00 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh stop
Jul 18 01:10:12 HOSTNAME ReportingEngine_Backup: Successfully created backup for /var/netwitness/re-server
Jul 18 01:10:12 HOSTNAME ReportingEngine_Backup: Starting Reporting Engine Service
Jul 18 01:10:12 HOSTNAME systemd: Started Reporting Engine.
Jul 18 01:10:12 HOSTNAME reporting-engine.sh: Mon Jul 18 01:10:12 UTC 2022 Starting /opt/rsa/soc/reporting-engine/bin/reporting-engine.sh restart_blocking_mode
.
.
Tasks
There is a "schedule-standby-admin-data-sync" script on crontab of admin server as you can see below.
This script is performing backup in primary admin server and then copy it to secondary admin server every day.
The "schedule-standby-admin-data-sync" script just calls the nw-recovery-tool to do the backup of all components, reporting-engine being one of many.
Each component supplies a backup.sh script that the nw-recovery-tool calls.
Therefore "schedule-standby-admin-data-sync" script causes this issue.
Resolution
This is expected behavior so there is no workaround at this moment.You can ignore this alarm because the alarm will be automatically disappeared once reporting engine is fully restarted.
Approval Reviewer Queue
Technical approval queue