Skip to content
  • There are no suggestions because the search field is empty.

Recovering and using core dump files from RSA NetWitness Platform core services in version 11.3.2 and above

Issue

This article outlines how to provide stack traces to the RSA NetWitness Platform developers when services crash.


Resolution

Starting with version 11.3.2 of the RSA NetWitness Platform, core services check for the existence of core dump files on startup. If any are found in the service’s working directory (/var/netwitness/ /packetdb), the following procedure takes place:
  1. The core dump file is moved to a cores subdirectory for the associated database files. (e.g. /var/netwitness/ /packetdb/cores).
  2. An attempt is made to locate a usable GNU Debugger (gdb) executable. If one is found, a stack trace generated from the core dump will be written to a file alongside the core dump file named .log.
  3. The core dump file is compressed using LZ4 compression, which renames the file to end in .lz4.
  4. If the compressed file is still larger than 4 GB, the file will be split into partitions that are no larger than 4 GB. These partitions will be named *.001, *.002, etc.
Also note that files in the cores subdirectory older than 90 days will be deleted to conserve space.

Deliver a core dump file

If a core dump file has been requested, it is generally sufficient to deliver the compressed core dump file or its partitions. If the file has been partitioned, it is not necessary to reassemble it; particularly if doing so might complicate transport. However, it is important to ensure that all partitions are delivered to the requesting party.

Generate a stack trace from a compressed (and possibly partitioned) core dump file

To obtain a stack trace from a core dump file, the file must be reassembled—if it has been split into multiple partitions—and decompressed.
  1. If the core dump file has been partitioned, use the command line utility cat to reassemble the partitions:
    $ cat <filename>.lz4.001 <filename>.lz4.002 <filename>.lz4.003 > \
    <filename>.lz4
    The above example applies to a core dump file that has been partitioned into three parts; though this may vary depending on the size of the original compressed file. It is important to ensure that all partitions are included when reassembling the compressed file.
     
  2. Use unlz4 to decompress the file:
    $ unlz4 <filename>.lz4
  3. Finally, generate a stack trace using gdb:
    $ gdb --batch -ex "thread apply all bt full" <service-executable> \
    <core-dump-file> &> stack_trace.txt

Notes

Getting unlz4

unlz4 is available from the CentOS repositories but may not be installed on appliances by default. It can be installed with yum using the command below:
$ yum install lz4

Getting gdb

While gdb is available from the CentOS repositories, a version that may have better compatibility with the compiler used to build the RSA NetWitness Platform core services is available in the Software Collections repositories. These can be made accessible by installing the repository configuration:
$ yum install centos-release-scl-rh
Following this, install the devtoolset-7-gdb package:
$ yum install devtoolset-7-gdb
The devtoolset-7 tools (including gdb) can be placed in the PATH of the current shell with the command below.
$ source /opt/rh/devtoolset-7/enable

Product Details

This article applies to version 11.3.2 and above of the RSA NetWitness Platform.


Approval Reviewer Queue

RSA NetWitness Suite Approval Queue