Skip to content
  • There are no suggestions because the search field is empty.

Reporting Engine will not start after upgrading

Issue

Unable to start the Reporting Engine service after upgrading.

Check the latest reporting-engine.sh_*.log on the server running the Reporting Engine Service
# cd /var/netwitness/re-server/rsa/soc/reporting-engine/logs/
# ls reporting-engine.sh_*.log -altr | tail -n1
-rw-r--r--. 1 rsasoc rsasoc 12746 Mar 8 05:49 reporting-engine.sh_20160308.log
reporting-engine.sh_20160308.log contains the following 'Fatal Error':
2016-03-02 06:03:46.640 : Log file will be located at /home/rsasoc/rsa/soc/reporting-engine/logs/reporting-engine.log
[ Fatal Error] :190:82: The entity name must immediately follow the '&' in the entity reference.
Reporting Engine version 10.5.1.0.5405-5
Initializing authorization cache.
[ Fatal Error] :190:82: The entity name must immediately follow the '&' in the entity reference.
2016-03-02 06:03:53.562 : Unable to start reporting-engine
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'Upgrade_RE10_3': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field:
---output snipped ----
Caused by: com.rsa.soc.re.exception.ReportingException: Error initializing authorization cache
at com.rsa.soc.re.authorization.DelegatingRoleManager.initializeCache(DelegatingRoleManager.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:344)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:295)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
... 46 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 190; columnNumber: 82; The entity name must immediately follow the '&' in the entity reference.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
at com.rsa.netwitness.carlos.security.authorization.BalanaXacmlAuthorizationManager.readPolicyDocument(BalanaXacmlAuthorizationManager.java:778)
at com.rsa.netwitness.carlos.security.authorization.BalanaXacmlAuthorizationManager.getRolePermissions(BalanaXacmlAuthorizationManager.java:335)
at com.rsa.netwitness.carlos.security.authorization.BalanaXacmlAuthorizationManager.getRolePermissions(BalanaXacmlAuthorizationManager.java:54)
at com.rsa.soc.re.authorization.DelegatingRoleManager.initializeCache(DelegatingRoleManager.java:68)
reporting-engine.log contains the following:
2016-03-02 06:03:53,544 [main] WARN org.springframework.context.support.ClassPathXmlApplicationContext - Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'Upgrade_RE10_3': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException:
Could not autowire field: com.rsa.soc.re.authorization.AuthorizationUtil com.rsa.soc.re.upgrade.Upgrade_RE10_3.authorizationUtil; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'authorizationUtil': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.rsa.soc.re.authorization.DelegatingRoleManager com.rsa.soc.re.authorization.AuthorizationUtil.roleManager;
nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'roleManager': Invocation of init method failed; nested exception is com.rsa.soc.re.exception.ReportingException: Error initializing authorization cache
---output snipped ----
Caused by: com.rsa.soc.re.exception.ReportingException: Error initializing authorization cache
at com.rsa.soc.re.authorization.DelegatingRoleManager.initializeCache(DelegatingRoleManager.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:344)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:295)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
... 46 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 190; columnNumber: 82; The entity name must immediately follow the '&' in the entity reference.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
at com.rsa.netwitness.carlos.security.authorization.BalanaXacmlAuthorizationManager.readPolicyDocument(BalanaXacmlAuthorizationManager.java:778)
at com.rsa.netwitness.carlos.security.authorization.BalanaXacmlAuthorizationManager.getRolePermissions(BalanaXacmlAuthorizationManager.java:335)
at com.rsa.netwitness.carlos.security.authorization.BalanaXacmlAuthorizationManager.getRolePermissions(BalanaXacmlAuthorizationManager.java:54)
at com.rsa.soc.re.authorization.DelegatingRoleManager.initializeCache(DelegatingRoleManager.java:68)
... 53 more

Cause

An issue with XML parsing causes a failure when initializing authorization cache.

This may be an issue with the parsing of XML files located in /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/permissions or /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/roles
# cd /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/permissions

# ll
total 2052
-rw-r--r--. 1 rsasoc rsasoc 63445 Mar 2 05:30 2afdb232a2ebd58f8a59d9bfff6640f9
-rw-r--r--. 1 rsasoc rsasoc 704581 Mar 2 05:30 4b10734d7c577bcaf9eede1d1903ca91
-rw-r--r--. 1 rsasoc rsasoc 277360 Mar 2 05:30 5fa14cbfa74f43602f52b7b15336315a
-rw-r--r--. 1 rsasoc rsasoc 219231 Mar 2 05:30 6ed265322215d658d66cb3c4947ded36
-rw-r--r--. 1 rsasoc rsasoc 13948 Mar 2 05:30 73acd9a5972130b75066c82595a1fae3
-rw-r--r--. 1 rsasoc rsasoc 63217 Mar 2 05:30 89f7a0b767b1fc8242c2fa47ceda8186
-rw-r--r--. 1 rsasoc rsasoc 62989 Mar 2 05:30 b3c5827f54218753bb2c3338236446c2
-rw-r--r--. 1 rsasoc rsasoc 617425 Mar 2 05:31 c1ec1d62eafad32ca16fe4df49b9ca2f
-rw-r--r--. 1 rsasoc rsasoc 61145 Mar 2 05:30 f3c4832e162a31612a38d61237c69884
Check that the XML files can be parsed using xmllint:

All Files
# find /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/permissions -type f | xargs xmllint -format {}
Individual Files - Example below shows the same error as found in the Reporting Engine logs
# xmllint -format /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/permissions/f3c4832e162a31612a38d61237c69884
f3c4832e162a31612a38d61237c69884:1310: parser error : xmlParseEntityRef: no name
AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">Bad String &
Look at lines around the problem string:
# grep -B5 -A2 'Bad String &' /home/rsasoc/rsa/soc/reporting-engine/security-policy/permissions/f3c4832e162a31612a38d61237c69884
<Rule RuleId="Permission:RE_User:rule63" Effect="Permit" >
<Target>
<Resources>
<Resource>
<ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string"> Bad String & Here</AttributeValue>
<ResourceAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:1.0:resource:resource-id" DataType="http://www.w3.org/2001/XMLSchema#string"/>
</ResourceMatch>
Check how many files have this problem string:
# grep 'Bad String & Here' /home/rsasoc/rsa/soc/reporting-engine/security-policy/permissions/*

Resolution

Backup the permissions files and perform string replacement within all files.
  1. Make sure the Reporting Engine Service is stopped:
  • systemctl stop rsasoc_re
  • Note: The command will probably produce 'stop: Unknown instance:' as Reporting Engine service is already stopped due to fatal error previously noted
  1. Backup the files we are about to edit:
    • cd /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/permissions
      tar cvjf /root/RE_permissions_backup.tar.bz2 *
  2. Perform in-file string replacement:
    • sed -ri 's/Bad String & Here/Bad String and Here/g' /var/netwitness/re-server/rsa/soc/reporting-engine/security-policy/permissions/*
  3. Restart reporting engine service:
  • systemctl start rsasoc_re

Product Details

NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: Reporting Engine 
NetWitness Version/Condition: 11.x , 12.x
Platform: CentOS , AlmaLinux


Approval Reviewer Queue

Technical approval queue