Skip to content
  • There are no suggestions because the search field is empty.

Malware Analysis device is not processing events in RSA Security Analytics 10.3 SP3

Issue

Malware Analysis device is not processing events in RSA Security Analytics 10.3 SP3.
The /var/lib/netwitness/uax/logs/sa.log reporsts the following error message:  WARN com.netwitness.malware.server.scheduler.jobs.EventJob - Inbound event processing throttled, too many events queued

Resolution

The first step is to do a thread dump to identify why the malware device has stopped processing events.

Use the command.

           ps aux |grep java

root    16030  0.0  0.0  79016  2512 ?        Ss   08:10   0:00 sudo -u rsamalware /usr/bin/java -Xms3936M -Xmx5196M -Djetty.home=/var/lib/rsamalware/jetty -Djava.io.tmpdir=/tmp -Djava.library.path=/var/lib/rsamalware/lib -Dcom.rsa.netwitness.carlos.CONFIG_XML=true -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=/var/lib/rsamalware/spectrum/logs/rsaMalwareDeviceVM.log -XX:NewSize=400M -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+OptimizeStringConcat -XX:+AggressiveHeap -XX:+UseAdaptiveGCBoundary -XX:MaxGCPauseMillis=2000 -jar /var/lib/rsamalware/jetty/start.jar --pre=etc/jetty-logging.xml --daemon
498      16031 50.7 57.6 7270832 4644808 ?     Sl   08:10  70:22 /usr/bin/java -Xms3936M -Xmx5196M -Djetty.home=/var/lib/rsamalware/jetty -Djava.io.tmpdir=/tmp -Djava.library.path=/var/lib/rsamalware/lib -Dcom.rsa.netwitness.carlos.CONFIG_XML=true -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=/var/lib/rsamalware/spectrum/logs/rsaMalwareDeviceVM.log -XX:NewSize=400M -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+OptimizeStringConcat -XX:+AggressiveHeap -XX:+UseAdaptiveGCBoundary -XX:MaxGCPauseMillis=2000 -jar /var/lib/rsamalware/jetty/start.jar --pre=etc/jetty-logging.xml --daemon
root     21621  0.0  0.0   6380   724 pts/2    S+   10:28   0:00 grep java

Then run kill -3 on the lowest PID

        [root@NWAPPLIANCE16112 ~]# kill -3  16030

This will kill the rsaMalware process.

The Log file will be saved in /var/lib/rsamalware/spectrum/logs.

Zip up the entire directory with the command.

       tar zcvf /tmp/malwarelogs.tgz /var/lib/rsamalware/spectrum/logs

These file /tmp/malwarelogs.tgz should then be copied off the Malware device using for example WinSCP and sent to RSA Customer Support for analysis.

The second step is to remove the oldest entries from the file being scanned.

    stop rsaMalwareDevice;

    su postgres;

    psql spectrum;

    select * from positions order by position asc;

 

This will display the oldest entry

 

spectrum=# select * from positions order by positions asc;

 

    id     | position |           source            

-----------+----------+-----------------------------

946815590 |  4926537 | nws://QASPECTRUM4:56003/sdk

946815589 |  4926538 | nws://QASPECTRUM4:56003/sdk

946815588 |  4926539 | nws://QASPECTRUM4:56003/sdk

946815587 |  4926540 | nws://QASPECTRUM4:56003/sdk

946815586 |  4926541 | nws://QASPECTRUM4:56003/sdk

 

 

delete from positions where position='4926537';

\q

exit;

start rsaMalwareDevice;

 

Alternatively, you can clear the entire queue (depending on what your goal is) by doing:

      truncate table positions;

instead of the deletestatement.



Notes

You can also access the Malware Legacy UI which can assist with troubleshooting.

Go to https://malware.device.fqdn:8443 and login with the admin / netwitness credentials  and then click on System -> Health as shown in the screenshot below you can see what each thread is doing.

 Malware Analysis device is not processing events in RSA Security Analytics 10.3 SP3

Malware Analysis device is not processing events in RSA Security Analytics 10.3 SP3



Product Details

RSA Security Analytics
RSA Security Analytics 10.3.3
RSA Security Analytics Malware Analysis
INTERNAL ONLY !!!