Skip to content
  • There are no suggestions because the search field is empty.

Concentrator aggregation immediately stops after starting aggregation in RSA Security Analytics

Issue

Concentrator aggregation stops immediately after start aggregation. 

The following error message will be observed in /var/log/messages:
Dec 16 22:54:28 NWAPPLIANCE nw[39892]: [Aggregation] [failure] There was a problem at initialization for device
'127.0.0.1:56002'. Newest remote session was 524,505 but last local session was 6,845,194,697. Consumption has been stopped.

 

Cause

Possible cause may include but not limited to, the appliance is RMAed and old data (metadb,sessiondb,index,etc) are restored from backup.
The problem is the Concentrator has very large session numbers in its old database and the newly consumed sessions are much lower and it seems to be confusing the issue.

Workaround

The easiest way to work around this is to change the Log Decoder or Decoder service name.
  1. In the Packet/Log Decoder Explore view, navigate to sys -> config -> service.name.override and add the new hostname to its value.
  2. Restart the nwlogdecoder /nwdecoder service.
  3. Restart Concentrator aggregation.
  4. Verify that the errors in /var/log/messages are no longer occurring.
  5. The old data should be populated under the old Packet/Log Decoder name in investigations.
  6. The new data will be populated under new Packet/Log Decoder name in investigations.
Another workaround is to delete the entries specific to the Packet or Log Decoder from the NwConcentrator.cfg file
  1. Stop the nwconcentrator service by SSHing to the stop nwconcentrator
  2. Make a backup of the /etc/netwitness/ng/NwConcentrator.cfg file
  3. Edit the /etc/netwitness/ng/NwConcentrator.cfg to remove the two entries where the Decoder is mentioned. This is highlighted below.
  4. Start the nwconcentrator service with start nwconcentrator
  5. Readd the Log decoder or decoder via the GUI
Example of changes made in NwConcentrator.cfg file.

The decoder entry needs to be deleted from
  1. Under: 
    <folder instance="folder" name="recovery" prettyName="recovery">
  2. Under 
    <folder instance="folder" name="devices" prettyName="devices">
Here it the file with the lines to be deleted in RED. In this case, we are deleting a packet decoder 192.168.123.2 on port 56004
 
<?xml version="1.0" encoding="UTF-8"?>
<root date="2017-Mar-06 10:43:54" doc-version="1" nw-version="10.6.3.0">
<folder instance="folder" name="concentrator" prettyName="concentrator">

... snip ...

<folder instance="folder" name="recovery" prettyName="recovery">
<folder instance="folder" name="192.168.123.242:50002" prettyName="192.168.123.242:50002">
<config getRoles="" instance="config" maxLength="2048" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="192.168.123.249:50003" prettyName="192.168.123.249:50003">
<config getRoles="" instance="config" maxLength="2048" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="192.168.123.2:50004" prettyName="192.168.123.2:50004">
<config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value="7463934:507114659-507139537"/>
</folder>
<folder instance="folder" name="192.168.123.2:56004" prettyName="192.168.123.2:56004">
<config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="192.168.123.3:50002" prettyName="192.168.123.3:50002">
<config getRoles="" instance="config" maxLength="2048" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="192.168.123.3:56002" prettyName="192.168.123.3:56002">
<config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="192.168.123.44:50002" prettyName="192.168.123.44:50002">
<config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="BROKER:50003" prettyName="BROKER:50003">
<config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value=""/>
</folder>
<folder instance="folder" name="NWAPPLIANCE9201:50003" prettyName="NWAPPLIANCE9201:50003">
<config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
</folder>
<folder instance="folder" name="broker:50003" prettyName="broker:50003">
<config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="malware:50003" prettyName="malware:50003">
<config getRoles="" instance="config" maxLength="4096" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value=""/>
</folder>
<folder instance="folder" name="nwappliance16112" prettyName="nwappliance16112">
<config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
</folder>
<folder instance="folder" name="nwappliance20886" prettyName="nwappliance20886">
<config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
</folder>
<folder instance="folder" name="packetconc:50005" prettyName="packetconc:50005">
<config getRoles="" instance="config" maxLength="4096" name="device.invalid.sessions" prettyName="device.invalid.sessions" setRoles="" value=""/>
</folder>
<folder instance="folder" name="sa" prettyName="sa">
<config getRoles="" instance="config" maxLength="2048" name="sessions.invalid" prettyName="sessions.invalid" setRoles="" value="1-2455681914"/>
</folder>
</folder>
<folder instance="folder" name="rules" prettyName="rules">

... snip ...
<folder instance="folder" name="correlation" prettyName="correlation">
<config getRoles="rules.manage" instance="config" maxLength="8192" name="0001" prettyName="0001" setRoles="rules.manage" </folder>
</folder>
</folder>
<folder instance="folder" name="devices" prettyName="devices">
<folder instance="folder" name="192.168.123.2:56004" prettyName="192.168.123.2:56004">
<folder instance="device" name="config" prettyName="config">
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="backup" prettyName="Backup" setRoles="concentrator.manage" value="no"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="name" prettyName="Name" setRoles="concentrator.manage" value=""/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="8192" name="options" prettyName="Options" setRoles="concentrator.manage" value=""/>
<config getRoles="concentrator.manage" instance="device.password.config" maxLength="255" name="password" prettyName="Password" setRoles="concentrator.manage" value="74BFC2D02C282038C1090DD08E1A4402"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="reconnect" prettyName="Reconnect" setRoles="concentrator.manage" value="yes"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="ssl" prettyName="SSL" setRoles="concentrator.manage" value="yes"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="username" prettyName="Username" setRoles="concentrator.manage" value="admin"/>
</folder>
</folder>
<folder instance="folder" name="192.168.123.3:56002" prettyName="192.168.123.3:56002">
<folder instance="device" name="config" prettyName="config">
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="backup" prettyName="Backup" setRoles="concentrator.manage" value="no"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="name" prettyName="Name" setRoles="concentrator.manage" value=""/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="1024" name="options" prettyName="Options" setRoles="concentrator.manage" value=""/>
<config getRoles="concentrator.manage" instance="device.password.config" maxLength="255" name="password" prettyName="Password" setRoles="concentrator.manage" value="74BFC2D02C282038C1090DD08E1A4402"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="reconnect" prettyName="Reconnect" setRoles="concentrator.manage" value="yes"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="ssl" prettyName="SSL" setRoles="concentrator.manage" value="yes"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="username" prettyName="Username" setRoles="concentrator.manage" value="admin"/>
</folder>
</folder>
<folder instance="folder" name="192.168.123.44:50002" prettyName="192.168.123.44:50002">
<folder instance="device" name="config" prettyName="config">
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="backup" prettyName="Backup" setRoles="concentrator.manage" value="no"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="name" prettyName="Name" setRoles="concentrator.manage" value=""/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="8192" name="options" prettyName="Options" setRoles="concentrator.manage" value=""/>
<config getRoles="concentrator.manage" instance="device.password.config" maxLength="255" name="password" prettyName="Password" setRoles="concentrator.manage" value="74BFC2D02C282038C1090DD08E1A4402"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="reconnect" prettyName="Reconnect" setRoles="concentrator.manage" value="yes"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="ssl" prettyName="SSL" setRoles="concentrator.manage" value="no"/>
<config getRoles="concentrator.manage" instance="device.config" maxLength="255" name="username" prettyName="Username" setRoles="concentrator.manage" value="admin"/>
</folder>
</folder>
</folder>
</folder>
<folder instance="folder" name="database" prettyName="database">
<folder instance="folder" name="config" prettyName="config">

... snip ...

</folder>
</folder>
</root>

Resolution

One option is to change the hostname and reboot the Packet/Log Decoder, at which point the Concentrator will recognize the Decoder as a new one and accept the smaller session.

Note: It is first worth entering the existing hostname of the Packet or Log Decoder in the service.name.override and following the steps below as this will avoid the need to actually change the hostname of the device.

Notes

The old log will be rolled out when DB maximum size is reached.


Product Details

RSA Product Set: Security Analytics
RSA Product/Service Type: Concentrator
RSA Version/Condition: 10.4.0.2, 10.X
Platform: CentOS
O/S Version: EL6

Approval Reviewer Queue

RSA NetWitness Suite Approval Queue