.png?width=180&height=32&name=NW%20Logo%20(2).png){kind=small}
Issue
The following symptoms are seen:
- When investigating, the meta is shown when but when you click on the green number no sessions are shown.
- A large number of unique values exist in the meta key
- In /var/log/messages on the concentrator the following message will be seen:
[root@NW11-NETWORK-HYBRID ~]# grep "has reached max capacity" /var/log/messages
Feb 10 16:05:23 NW11-NETWORK-HYBRID NwConcentrator[1334]: [Index] [warning] Index key session.split has reached max capacity of 100 values and will ignore new values for this slice.
Feb 14 17:20:05 NW11-NETWORK-HYBRID NwConcentrator[1396]: [Index] [warning] Index key session.split has reached max capacity of 100 values and will ignore new values for this slice. - In this case the meta key session.split is full and cannot contain any more values in this index slice as the maximum number of unique values has been reached.
Workaround
Health and Wellness provides an easy way to determine if a metakey is overflown (exceeded the unique values allocated per Index slice) and will therefore cause investigation problems. Once the metakey overflow is determined via H&W, adjustments can be made to the index size ValueMax or to the parser.
Here is an example of what a H&W Alarm looks like when a metakey is overflown:
In this example, it is showing that the "session.split" metakey is the one that has overflown (exceeded the allocated unique values per Index slice).
From the H&W Alarm details, a more verbose explanation:
Resolution
The issue with meta being seen in the session but not being able to investigate on, is often due to the number of unique meta key values filling up beyond the "Value Max" parameter specified in the index, which resembles this in the default index-concentrator.xml
[root@NW11-NETWORK-HYBRID ~]# grep "event.desc" /etc/netwitness/ng/index-concentrator.xml
<key description="Event Description" name="event.desc" format="Text" singleton="true" level="IndexValues" valueMax="1000000" defaultAction="Open"/>
The Value Max parameter in the index-concentrator.xml or index-concentrator-custom.xml determines the number of unique values that the meta key can hold per index slice. By default an index slice gets created every 8 hours, or if it has been configured every 600 Million Sessions.
If for example your meta key holds 100000 values, then additional unique values will not be indexed, which means that you cannot click through to them when investigating.
The values can be viewed though when you look at the event session view.
The solution to this it one of the following:
1) If the values stored in the metakey are as desired, then increase the value max setting in the index-concentrator-custom.xml for this meta key. Restart the concentrator service for the changes to then take effect. More information: https://community.netwitness.com/s/article/IndexCustomization
2) If the values in the metakey are not desired - perhaps there is a lot of misparsed information or other information being stored in the this metakey, then the best approach is to identify the source of this misparsed information and then correct it. This might for example involve updating a parser. An example of misparsed information can be seen here:
Here we see that the event description key is holding a lot of unique values, that are essentially similar. The values are of the following form:
The parser should be updated so that these messages are broken down into constituent parts.
A more appropriate event description would perhaps be "no space left on device".
This would then prevent the event description meta key filling up with values that we are not interested in.
Notes
More information on customizing the index can be found here:
https://community.netwitness.com/s/article/IndexCustomization
For an explanation of index slices see the topic of Index Saves here:
https://community.netwitness.com/s/article/How-to-monitor-if-a-meta-index-key-is-full-in-the-RSA-NetWitness-Platform
More information on adding custom metakeys can be found here:
https://community.netwitness.com/s/article/How-to-add-custom-meta-keys-in-RSA-NetWitness-Platform
Product Details
NetWitness Product Set: NetWitness Logs & Network
NetWitness Product/Service Type: Concentrator, Retention Log Decoder
NetWitness Version/Condition: 11.x , 12.x
Platform: CentOS , AlmaLinux
Summary
When investigating, the meta is shown when but when you click on the green number no sessions are shown.
Approval Reviewer Queue
Technical approval queue