Event Analysis can not load data from all core devices in RSA NetWitness Logs & Network 11.1.0.0

Issue

Event Analysis cannot load data from all core devices.
Customer restarted jetty and investigate service, but it could not fix this issue.
From only Event analysis page, it cannot load data from all core devices but navigate and events can load data from the core devices.

The rpm installed is

 rsa-nw-investigate-server-11.1.0.0-180222160922.5.6cc8403.el7.centos.noarch

Following messages occurred in investigate-server.log

 2018-09-20 04:06:01,309 [ clientInboundChannel-190353] ERROR DataAccess|Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
 
2018-09-20 04:06:01,309 [ clientInboundChannel-190353] WARN DataAccess|Failed to retrieve data
 
java.util.concurrent.CompletionException: org.apache.mina.core.RuntimeIoException: Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
 
 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
 
 at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
 
 at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
 
 at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614)
 
 at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultInvestigateRepository.findLanguages(DefaultInvestigateRepository.java:165)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultInvestigateRepository$$FastClassBySpringCGLIB$$6ff03bc6.invoke(<generated>)
 
 at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
 
 at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
 
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 
 at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
 
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 
 at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultInvestigateRepository$$EnhancerBySpringCGLIB$$66a6ccb.findLanguages(<generated>)
 
 at com.rsa.asoc.nw.investigate.server.service.DefaultInvestigateService.findLanguages(DefaultInvestigateService.java:85)
 
 at com.rsa.asoc.nw.investigate.server.service.DefaultInvestigateService.findAliases(DefaultInvestigateService.java:121)
 
 at com.rsa.asoc.nw.web.common.service.DefaultWebSocketMessagingService.message(DefaultWebSocketMessagingService.java:96)
 
 at com.rsa.asoc.nw.investigate.server.web.socket.InvestigateController.findAliases(InvestigateController.java:122)
 
 at com.rsa.asoc.nw.investigate.server.web.socket.InvestigateController$$FastClassBySpringCGLIB$$73853dbe.invoke(<generated>)
 
 at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
 
 at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
 
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
 
 at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
 
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 
 at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
 
 at com.rsa.asoc.nw.investigate.server.web.socket.InvestigateController$$EnhancerBySpringCGLIB$$d8cd6791.findAliases(<generated>)
 
 at sun.reflect.GeneratedMethodAccessor313.invoke(Unknown Source)
 
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 
 at java.lang.reflect.Method.invoke(Method.java:498)
 
 at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
 
 at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
 
 at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMatch(AbstractMethodMessageHandler.java:504)
 
 at org.springframework.messaging.simp.annotation.support.SimpAnnotationMethodMessageHandler.handleMatch(SimpAnnotationMethodMessageHandler.java:497)
 
 at org.springframework.messaging.simp.annotation.support.SimpAnnotationMethodMessageHandler.handleMatch(SimpAnnotationMethodMessageHandler.java:87)
 
 at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMessageInternal(AbstractMethodMessageHandler.java:463)
 
 at org.springframework.messaging.handler.invocation.AbstractMethodMessageHandler.handleMessage(AbstractMethodMessageHandler.java:401)
 
 at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:135)
 
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 
 at java.lang.Thread.run(Thread.java:748)
 
Caused by: org.apache.mina.core.RuntimeIoException: Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
 
 at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:220)
 
 at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:115)
 
 at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:97)
 
 at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
 
 at com.rsa.asoc.transport.nw.session.NextgenConnection.getSocketConnector(NextgenConnection.java:368)
 
 at com.rsa.asoc.transport.nw.session.NextgenConnection.open(NextgenConnection.java:108)
 
 at com.rsa.asoc.transport.nw.session.NextgenConnection.connect(NextgenConnection.java:492)
 
 at com.rsa.asoc.nw.investigate.server.handler.DefaultNextgenConnectionHandler.getNextgenConnection(DefaultNextgenConnectionHandler.java:53)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultNextgenRepositoryEngine.send(DefaultNextgenRepositoryEngine.java:68)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultNextgenRepositoryEngine.send(DefaultNextgenRepositoryEngine.java:51)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultNextgenRepositoryEngine.send(DefaultNextgenRepositoryEngine.java:44)
 
 at com.rsa.asoc.nw.investigate.server.domain.repository.DefaultInvestigateRepository.findLanguages(DefaultInvestigateRepository.java:163)
 
 ... 34 common frames omitted
 
Caused by: java.lang.reflect.InvocationTargetException: null
 
 at sun.reflect.GeneratedConstructorAccessor67.newInstance(Unknown Source)
 
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 
 at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:198)
 
 ... 45 common frames omitted
 
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
 
 at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:64)
 
 ... 49 common frames omitted
 
Caused by: java.io.IOException: Too many open files
 
 at sun.nio.ch.IOUtil.makePipe(Native Method)
 
 at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
 
 at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
 
 at java.nio.channels.Selector.open(Selector.java:227)
 
 at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:62)
 
 ... 49 common frames omitted

Cause

I have looked into the message log file and found below error

 Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: 2018-10-11 06:15:18.497 ERROR 29268 ? [dChannel-192707] o.a.m.c.service.SimpleIoProcessorPool : Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
 
Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: java.lang.reflect.InvocationTargetException: null
 
Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: at sun.reflect.GeneratedConstructorAccessor67.newInstance(Unknown Source)

It is due to Caused by: java.io.IOException: Too many open files
It looks that it is running out of file descriptors limit defined.

 Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
 
Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:64)
 
Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: ... 48 common frames omitted
 
Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: Caused by: java.io.IOException: Too many open files
 
Oct 11 06:15:18 NWAPPLIANCE31005 investigate-server.jar: at sun.nio.ch.IOUtil.makePipe(Native Method)

Workaround

Increase the value of ulimit in the file /opt/rsa/jetty9/bin/jetty.sh from 65536 to 131072 and restart the jetty and investigate-server service.

1. Change the value from 65536 to 131072 in jetty.sh
ulimit -Hn 131072
ulimit -Sn 131072

2. Restart following services.
#systemctl restart jetty.service
#systemctl restart rsa-nw-investigate-server.service

Product Details

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.1.0.0

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue