.png?width=180&height=32&name=NW%20Logo%20(2).png){kind=small}
How to analyze packet drops on RSA Security Analytics Packet Decoders
Issue
How to analyze packet drops on RSA Security Analytics Packet Decoders.RSA Security Analytics Packet Drop Analysis Tools.
Resolution
Installing and Using the Packet Drop Analyzer Tool
The SA-Core team developed a web-based tool to query and view packet drops and analyze decoder stats.
This tool queries the statdb files and log files on an appliance. The statdb (.statdb in /var/netwitness/decoder/statdb) can be copied from the customer environment and placed on a test or lab host to run these queries to check and analyze packet drops. However, you will not have the customer logs associated with the statsdb information.
If the customer agrees to perform analysis on their decoders, then you can copy the above files to customer environment and analyze packet drops directly on their Decoder. If you run on these tests on the customer?s Decoder you would also have relevant logs.
Three files must be deployed to a Decoder to use this tool Copy the following three files to /etc/netwitness/ng/app folder on the target appliance.
reports.html
sessions.html
common.js
It is not necessary to stop or start any services on the target Decoder when installing these files.
Once these files are on the Decoder, use the a web browser to query for drops.
The tool scans the statsdb database and gatherers packet drops and provides links to display graphs for stats when packet drops are detected with the following URL:
If using SSL: https://
If not using SSL: http://
Report URL optional parameters to specify time and other options:
time1 - beginning time (e.g. 2014-04-01 00:00:00), requires time2
time2 - end time (e.g. 2014-04-01 01:00:00), requires time1
analysis=true - return analysis for time range, requires time1 and time2. (The default value for analysis is false.)
The date format for reports must be specified in YYYY-MM-DD. You must only use digits at this time. If you use any other format, the entire range of stats will be queried.
Examples:
http://10.x.x.x:50104/sdk/app/reports?time1=2014-08-25%2000%3A00%3A01%3A22%20&time2=2014-08-25%2023%3A59%3A59
http://10.x.x.x:50104/sdk/app/reports?time1=2014-08-25%2000%3A00%3A01%3A22%20&time2=2014-08-25%2023%3A59%3A59&analysis=true
This tool also displays logs for the selected time range and for the packet drop period (5 min around packet drop.). This should make it easy to correlate logs and stats to identify the cause for packet drops.
To view and analyze a single stat use http://
statistic ? provide the full statistic path to one statistic (e.g. /decoder/stats/capture.packet.rate)
time1 - beginning time (e.g. 2014-Apr-01 00:00:00 or 2014-AUG-25) requires time2
time2 - end time (e.g. 2014-Apr-01 01:00:00 or 2014-AUG-25), requires time1
realtime - Number of seconds of data to initialize graph with (e.g. 60)
The date format for sessions may be specified in YYYY-MM-DD or in YYY-MMM-DD. You may use either digits or alpha characters at this time. For some reason, sessions.html converts alpha characters to digits before process while reports.html does not perform this conversion at this time.
Examples:
http://10.x.x.x:50104/sdk/app/sessions?/decoder/stats/capture.dropped&realtime=60
http://10.x.x.x:50104/sdk/app/sessions?/decoder/stats/capture.packet.rate&realtime=60
Note: The session query does not provide a progress indicator. You have no way of knowing your query is executing.
Note: The session query can take considerable time to execute so use patience for queries longer than a few minutes.
Notes
Note: (INTERNAL ONLY!!!)
The following procedure should be executed before escalating a case to CE or Development:
1. Collect information about the customer deployment environment and how the decoder is being used. Stand alone or hybrid? How many aggregators? Is there a malware/spectrum or other third party service that is requesting content from the decoder?
2. Collect a tech dump using the latest nwtech.sh that includes logs covering at least one representative "drop event"
3. Collect a stat history that spans at least one drop event showing normal system state preceding the drop event and recovery from the drop event. Please see Using the 9.8 Stats Database for how to collect stats and convert them to csv. At a minimum, /sys/stats*, /decoder/stats/*, /decoder/parsers/stats/*, and /database/stats/* should be included.
4. Use the collected stats to produce graphs of the relevant stats as demonstrated in the attached spreadsheet.
StatHist_Jun_10_17_55_to_18_00.xlsx
5. Extract the decoder logs spanning the event looking specifically for logged events corresponding to the beginning of the event.
6. Refer to additional information on interpreting the stats and logs as it is provided.
7. If the analysis of the decoder logs and statistics so far has not presented a course of action, include the tech dump, spreadsheet, extracted logs and any additional analysis/observation in the JIRA ticket.
8. If subsequent events are reported, include logs and statHist covering the event. An additional tech dump is could be considered optional, but should be collected if configuration changes have been made.
Review the following Engineering Wiki articles for additional information:
https://wiki.na.rsa.net/display/CNEX/Troubleshooting+Packet+Drops
https://wiki.na.rsa.net/display/CNEX/Using+the+9.8+Stats+Database
Internal Comments
UserName:saxonj9/8/2014 4:36:28 PM - Initial Comment
Reformatted and expanded article. The engineering Jira incorrectly states a lot of details expecially on time1 & time2 syntax. The tools can be given to the customer as needed but the document should be kept internal and the tool installed with our assistance until further notice.
UserName:shurtj
9/17/2014 5:20:09 PM - Technical Review
Article was technically reviewed and published by Sergio Almeida. Made minor changes to article to abide by Primus best practices.
Product Details
RSA Security AnalyticsRSA Security Analytics 10.3.4 and below
RSA Security Analytics 10.4
RSA Security Analytics Decoder
INTERNAL ONLY!!!