Centralized Backup and Restore of NetWitness Version 11.2+ (A Wrapper Script for NRT)

Updated for Version 12.5.x (backward compatible to 11.x)

Scenario -

Need to remotely backup your NetWitness hosts to a central location, to satisfy Disaster Recovery Requirements, perform a Tech Refreshes, or to be prepared for RMA replacement of a device.

Solution – A Wrapper for NRT

Building off the framework of the original nw-backup scripts written for 10.x backup/restore and migration to 11.x, a new set of version 11/12 scripts has been written as a "wrapper" to the built in NetWitness Recovery Tool (NRT) functionality of NetWitness since version 11.2 was released. 

(Please note that this is not an officially supported solution by NetWitness Support, but can be used by customers as a possible backup solution, at their own risk)

Overview –

The solution consists of 5 scripts (all run from the NW Admin server (node-zero)), supporting files for custom feed backup fix and an example nw-base.nrt file (note: The '11' has been removed from the script names since this can support versions 11 or 12):

• get-all-systems.sh – Generates an inventory file of all hosts in the NW server deployment.

• ssh-propagate.sh   – Generates an ECDSA key pair on the NW Server and propagates the public key to all hosts.
• nw-backup.sh       – The main backup script.
• nw-restore.sh      – Restore script for RMA/Tech refresh.
• cf-fix.sh          – Custom feed backup fix (used for applying fix to Warm Standby Server).
• feedfix folder     – Contains the backup.sh and restore.sh scripts to support the backup of custom
                       feed files on the admin server.
• nw-base.nrt        – An NRT script to add custom files to the backup directory as part of the backup process.

Process Walkthrough –

Copy the attached zip file to the NW Admin Server host (node-zero) and unzip:

mkdir /root/scripts

unzip nw-backup.zip -d /root/scripts

cd /root/scripts

chmod +x *.sh

Step 1: Run get-all-systems.sh

./get-all-systems.sh

Usage:

 ./get-all-systems.sh [ -u ][ -p ][-d /home/path ][ -b ][-n ][-h]

Configuration Options

Note: All command-line options are optional.

      With no options selected, the script will use options set within the file itself.

-b : Path to store the all-systems file and logs. Default:(/var/netwitness/nw-backup)

-u     : User acct to use for non-root SSH access to hosts during backups.  Default:(root)

-p     : Password for   Default: (ask)

-g         : Alternate Primary Group for entered username. Default: (username)

-d   : Base home directory path (/home) for Default:(/var/netwitness/nw-backup)

-n           : Skip check for new systems if current new-systems is less than old.

-h                : Print this help information.

Run on the NW Admin Server.  Creates the /var/netwitness/nw-backup directory (or the directory passed with the -b option), then using a combination of mongo and salt queries, will create the all-systems file in that directory.  The all-systems file is used by the other scripts , with entries that contain the following in comma separated format:

DeviceType,Hostname,IPAddress,MinionID,SerialNumber

DeviceType   = NRT Category Type (i.e. AdminServer,Broker,Concentrator,Decoder, etc.)

Hostname     = Short hostname (not FQDN)

IPAddress    = Management Interface IP address of host

MinionID     = Unique Salt MinionID of host

SerialNumber = Device Serial Number for Reference and Support

Example:

AdminServer,nw-admin,192.168.1.129,70f95dc0-3cb6-4fd4-b9f2-ac923d0ba594,PK10T51

ESAPrimary,nw-esa,192.168.1.131,a598cb6b-4bd2-4ba2-af6a-79df3dab35e6,R9L8LNM

Broker+Search,nw-broker,192.168.1.130, 2a83b597-7970-4872-b76f-109cb591fa90,CSZ77X2

LogHybrid,nw-loghyb,192.168.1.133,87fc872c-68e3-45e3-9108-e30f847dc14e,PK10T0A

Malware,nw-malware,192.168.1.132,2c98e425-57a0-47d2-82d7-15795a6165f5,R90BCFWP

NetworkHybrid,nw-nethyb,192.168.1.134,9a99294e-3889-48b0-9555-11d3c21e2018,R90218K6

The script is designed to run from the cron on a regular basis (if you are in a dynamic environment were systems are added/removed on a regular basis), it has a 30 second timeout on the one question it asks.

If changes to the environment are detected,  generation of new-systems and/or old-systems files  will occur.  The new-systems file can be used by other scripts for running specific targeted actions against the newly installed hosts.  If a system is “offline” or has been removed from the UI, the old-systems file will have the entries for those hosts, so information about them is not lost.

Step 2: Run ssh-propagate.sh

./ssh-propagate.sh

Configuration options

Note: All command-line options are optional.

  Run with no options, script uses the /var/netwitness/nw-backup/all-systems file to target

  all hosts and copy the root users ecdsa-521 bit public ssh key to all hosts

Usage:

./ssh-propagate.sh [ -u ] [-c] [ -b ] [ -t  ]

-b       : path to the location of all-systems file. Default: (/var/netwitness/nw-backup)

-u   : User to propagate keys to on all nodes (must exist on NW Server). Default: (root)

-g      : Group associated with . Default: (username)

-c             : Used with -u [ -g ] for non-root user, creates user on remote hosts.

-t : Target hosts (new-systems or anything grep-able from all-systems. Default: (ALL)

Run on the NW Admin Server.  Performs the following (depending on options):

• Verifies the designated user account already exists on the NW Server
• Updates the /etc/hosts file with host shortnames using entries from all-systems file and hosts.custom file (if it exists in the same directory as the script)
• Generates an ecdsa-521bit  ssh key for root (or the specified user) if it does not exist
• Iterates through the target list (default is ALL hosts in the all-systems file) and performs the following:
  • Tests ssh connectivity to the host (host responding on port 22)
  • Verifies the user exists on the remote host, if not and the -c option is selected
    • Creates remote user acct
    • Sets password to same as user acct on NW server
  • Tests SSH key authentication to host as the user specified, if auth fails:
    • Copies the user ecdsa public key string to the ~/.ssh/authorized_keys file
    • Adds the target host fingerprint to the user's ~/.ssh/known_hosts file on the NW server
    •  Verifies ssh connectivity via ssh-key authentication

Step 3: Modify the nw-base.nrt file

vi /root/scripts/nw-base.nrt

The default /etc/netwitness/recoverytool/nw-base.nrt file, distributed file with 11.x systems, only contains the following entries:

name nw-base

directory /etc/netwitness/platform/nodeinfo

file /etc/machine-id

# unmanaged files

stash /etc/fstab

stash /etc/hosts

stash /etc/sysconfig/iptables

stash /root/.ssh

# for azure

stash /etc/krb5.conf

stash /etc/logrotate.d/waagent.logrotate

stash /etc/mdadm.conf

stash /etc/waagent.conf

# only for nrt mode, run dnsclient to update the hosts configuration
after-import if [ "${NRT_MODE}" != 'standby' ]; then chef-client --config /var/lib/netwitness/config-management/client.rb --logfile /var/log/netwitness/config-management/chef-client.log --log_level info --runlist 'recipe[nw-dns-client]' > /dev/null 2>&1; fi

The “STASH” entries are NOT restored during an NRT import (recovery), but are available for reference in the /var/netwitness/backup/unmanaged folder, after the import.   The included nw-base.nrt file has an expanded list of stash files and directories to include files used at several customer installations.  To Fully restore functionality, these files need to be available after the restore. 

Extended nw-base.nrt supplied with the scripts:

name nw-base

directory /etc/netwitness/platform/nodeinfo

file /etc/machine-id

# unmanaged files

stash /etc/fstab

stash /etc/hosts

stash /etc/resolv.conf

stash /etc/nsswitch.conf

stash /etc/passwd

stash /etc/shadow

stash /etc/group

stash /etc/sudo.conf

stash /etc/sudoers

stash /etc/sudoers.d

stash /etc/exports

stash /etc/krb5.conf

stash /etc/nfs.conf

stash /etc/ntp.conf

stash /etc/rsyslog.conf

stash /etc/logrotate.conf

stash /etc/sysconfig/network

stash /etc/sysconfig/nfs

stash /etc/sysconfig/iptables

stash /etc/sysconfig/iptables.bak

stash /etc/sysconfig/iptables-config

stash /etc/crontab

stash /etc/sysconfig/network-scripts/ifcfg-em1

stash /etc/sysconfig/network-scripts/ifcfg-em2

stash /etc/sysconfig/network-scripts/ifcfg-em3

stash /etc/sysconfig/network-scripts/ifcfg-em4

stash /etc/cron.hourly

stash /etc/cron.daily

stash /etc/cron.weekly

stash /etc/cron.daily

stash /etc/pam.d/netwitness

stash /etc/pam.d/securityanalytics

stash /etc/logrotate.d

stash /etc/logstash

stash /etc/multipath.conf

stash /etc/lvm

stash /etc/raddb

stash /etc/rsyslog.d

stash /etc/snmp

stash /etc/ssh

stash /var/ace

stash /home

stash /root   (note: this backs up ALL file/folders in /root including /root/.ssh)

stash /var/netwitness/nw-backup/all-systems

# for azure

stash /etc/krb5.conf

stash /etc/logrotate.d/waagent.logrotate

stash /etc/mdadm.conf

stash /etc/waagent.conf

# only for nrt mode, run dnsclient to update the hosts configuration
after-import if [ "${NRT_MODE}" != 'standby' ]; then chef-client --config /var/lib/netwitness/config-management/client.rb --logfile /var/log/netwitness/config-management/chef-client.log --log_level info --runlist 'recipe[nw-dns-client]' > /dev/null 2>&1; fi

Edit the file to include any additional locations (files or directories) that contain customizations in your deployment, then save the file in the same directory as the nw-backup.sh script.  The backup script will verify the file on each system matches your modified file and if not, will automatically copy the modified file to each host before running NRT on that host.

Step 4: Run nw-backup.sh

./nw-backup.sh

Configuration Options:

Note: All command-line options are optional.

      With no options selected, the script backup ALL devices listed in the

      /var/netwitness/nw-backup/all-systems file(except any commented(#) out)

      and copy the backup files to the /var/netwitness/nw-backup/ / directory

      on the NW Server.

Usage:

./nw-backup.sh [-b ] [-m ] [-l ] [-p ] [-s ] [-d ] [-t ] [-u ] [-g ] [-U ] [-G ] -M -L -R -I

Options:

-b : path for NRT backup files. Default: (/var/netwitness/backup)

-m : remote transfer mode (scp or nfs). Default: (scp)

-p : Path on NW server for logs and location of all-systems

                                     file. Default: (/var/netwitnes/nw-backup)

-d : Path on destination server to move completed backup files to

                               via nfs or scp. Default: (/var/netwitness/nw-backup)

-s : Destination server IP address for storing completed backup files,

                        transferred via nfs or scp. Default: (NW Server IP)

-u : User acct for SCP transfers of completed backups, user must exist on all

               target systems and on Destination server. Default: (root)

-g : Group for assigning permission to files copied to remote host. Default (root)

-U : User acct on Remote system SCP transfers of completed backups, user must exist on
                  remote server adn have SSH-Key auth configured. Default: (root)

-G : Group associated with Remote SCP user for transfer of completed backups, group must
                   exist on remote server. Default (root)

-l : local mount point for NFS share. Default: (/mnt/backup)

-t : backup ONLY specific Target(s), (can be anything greppable from all-systems file)
              Default: (all)

               Special Targets:

               core (Broker, Concentrator, Decoder, LogDecoder, Archiver,

                        LogCollector(vlc), NetworkHybrid, LogHybrid)

               nonw (all devices except AdminServer)

               nwonly (AdminServer only)

               esaonly (all ESA devices only)

               endpoint (all endpoint devices EndpointHybrid, EndpointLogHybrid, Gateway)

Exclusions/Inclusions/Service Control:

-M : Exclude the Malware Analysis File Store.

-I : Include the Broker Index files for RMA/Tech Refresh situation. Default: (Exclude)

-L : DO NOT STOP LogCollector (nwlogcollector) service during backup. Default: (Stop Service)

-R : DO NOT STOP Reporting Engine (rsasoc_re) service during backup. Default: (Stop Service)

       Note: NRT normal operation stops these services, not stopping will affect:

             LogCollector - Will not have latest tracking data for some logs.

             Reporting Engine - Some live chart data and alert status data may be lost.

NOTES –

• If using SCP to a server other than the NW Admin Server, the copy uses the "-3" option and makes the transfer of the backup file via the NW Admin Server, so ONLY the NW Admin server needs to  have SSH key authentication configured to all hosts.
• Make sure the modified nw-base.nrt file is in the same directory you are running the nw-backup.sh script from, the script will hash that file, and verify that hash against the file on each server, if they do not match, it will copy the file in the script directory (where you ran the nw-backup.sh script from) to the remote host, before triggering the NRT backup on that  host. If you do not want to make any modifications to the default file, don’t include the file in the script run directory.
• Deploy Admin password is programmaticly called, so no exposure of password in the scripts
• The /var/netwitness/nw-backup/all-systems file can be used for a myriad of other scripting calls, or for targeting a specific type of host, especially when using “salt” commands.

Restore:  nw-restore.sh

• Follow the published backup/restore document for the version of NW you are running