RSA NetWitness Correlation Server Rules deployment times out due to low bandwidth
Issue
When Correlation Rules deployed, the deployment fails with a time-out error as below.
On NwServer
/var/lib/netwitness/uax/logs/sa.log:
2020-09-18 07:34:56,402 [pool-11-thread-1] ERROR com.rsa.smc.sa.esa.service.RuleSynchronizationService -
Request to correlation-server.45346322-5555-4b13-9d42-640b2f10dbf2./rsa/correlation/engine/ save timed out
2020-09-18 07:34:56,402 [pool-11-thread-1] ERROR com.rsa.smc.sa.esa.service.RuleSynchronizationService -
Request to correlation-server.c1236961-e1a5-4b13-9d42-640b2f10dbf2./rsa/correlation/engine/save timed outcom.rsa.asoc.launch.api.transport.client.RequestTimeoutException:
Request to correlation-server.c1236961-e1a5-4b13-9d42-640b2f10dbf2./rsa/correlation/engine/save timed out
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClientHelper.requestTimeoutException(AmqpTransportClientHelper.java:44)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSendAndReceive(AmqpTransportClient.java:115)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:69)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:68)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)
at com.sun.proxy.$Proxy434.save(Unknown Source) at com.rsa.smc.sa.esa.service.CorrelationServerService.deploy(CorrelationServerService.java:393)
at com.rsa.smc.sa.esa.service.DefaultRuleSynchronizationService.synchronize(DefaultRuleSynchronizationService.java:286)
at com.rsa.smc.sa.esa.web.controller.SynchronizationController.deploy(SynchronizationController.java:390)
at com.rsa.smc.sa.esa.web.controller.SynchronizationController.lambda$synchronizeGroup$0(SynchronizationController.java:373)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)2020-09-18 07:34:56,403 [pool-11-thread-1] INFO com.rsa.smc.sa.esa.service.RuleSynchronizationService -
Synchronization Failed for 10.10.10.10 - Temp
2020-09-18 07:34:56,402 [pool-11-thread-1] ERROR com.rsa.smc.sa.esa.service.RuleSynchronizationService -
Request to correlation-server.45346322-5555-4b13-9d42-640b2f10dbf2./rsa/correlation/engine/ save timed out
2020-09-18 07:34:56,402 [pool-11-thread-1] ERROR com.rsa.smc.sa.esa.service.RuleSynchronizationService -
Request to correlation-server.c1236961-e1a5-4b13-9d42-640b2f10dbf2./rsa/correlation/engine/save timed outcom.rsa.asoc.launch.api.transport.client.RequestTimeoutException:
Request to correlation-server.c1236961-e1a5-4b13-9d42-640b2f10dbf2./rsa/correlation/engine/save timed out
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClientHelper.requestTimeoutException(AmqpTransportClientHelper.java:44)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSendAndReceive(AmqpTransportClient.java:115)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:69)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:68)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)
at com.sun.proxy.$Proxy434.save(Unknown Source) at com.rsa.smc.sa.esa.service.CorrelationServerService.deploy(CorrelationServerService.java:393)
at com.rsa.smc.sa.esa.service.DefaultRuleSynchronizationService.synchronize(DefaultRuleSynchronizationService.java:286)
at com.rsa.smc.sa.esa.web.controller.SynchronizationController.deploy(SynchronizationController.java:390)
at com.rsa.smc.sa.esa.web.controller.SynchronizationController.lambda$synchronizeGroup$0(SynchronizationController.java:373)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)2020-09-18 07:34:56,403 [pool-11-thread-1] INFO com.rsa.smc.sa.esa.service.RuleSynchronizationService -
Synchronization Failed for 10.10.10.10 - Temp
Cause
This issue is due to low bandwidth between NwServer and Correlation Server.Please use below document to check bandwidth between two Servers.
How to check Network bandwidth in RSA NetWitness Platform between SA/Correlation server and Data Source concentrators
Note: Above document deals checking bandwidth between ESA and Concentrator. But, same steps can be used to check bandwidth between any two CentOS servers.
Workaround
Please follow below workaround until good bandwidth provided.- Please create multiple deployments.
- Segregate rules to deploy them to separate deployments.
- This will reduce the content to transfer between Servers and deployed successfully.
Resolution
Please communicate customer to work with internal network team to provide good bandwidth.
Product Details
RSA Product Set: RSA NetWitness PlatformRSA Product/Service Type: Event Stream Analysis
RSA Version/Condition: 11.X
Platform: CentOS
O/S Version: 7
Summary
This document outlines the procedure to fix time out issue while Correlation Rule deployment.
Approval Reviewer Queue
RSA NetWitness Suite Approval Queue