Need to remotely backup your NetWitness hosts to a central location, to satisfy Disaster Recovery Requirements, perform a Tech Refreshes, or to be prepared for RMA replacement of a device.
Building off the framework of the original nw-backup scripts written for 10.x backup/restore and migration to 11.x, a new set of version 11/12 scripts has been written as a "wrapper" to the built in NetWitness Recovery Tool (NRT) functionality of NetWitness since version 11.2 was released.
(Please note that this is not an officially supported solution by NetWitness Support, but can be used by customers as a possible backup solution, at their own risk)
The solution consists of 5 scripts (all run from the NW Admin server (node-zero)), supporting files for custom feed backup fix and an example nw-base.nrt file (note: The '11' has been removed from the script names since this can support versions 11 or 12):
Copy the attached zip file to the NW Admin Server host (node-zero) and unzip:
mkdir /root/scripts
unzip nw-backup.zip -d /root/scripts
cd /root/scripts
chmod +x *.sh
./get-all-systems.sh
Usage:
./get-all-systems.sh [ -u ][ -p ][-d /home/path ][ -b ][-n ][-h]
Configuration Options
Note: All command-line options are optional.
With no options selected, the script will use options set within the file itself.
-b : Path to store the all-systems file and logs. Default:(/var/netwitness/nw-backup)
-u : User acct to use for non-root SSH access to hosts during backups. Default:(root)
-p : Password for Default: (ask)
-g : Alternate Primary Group for entered username. Default: (username)
-d : Base home directory path (/home) for Default:(/var/netwitness/nw-backup)
-n : Skip check for new systems if current new-systems is less than old.
-h : Print this help information.
Run on the NW Admin Server. Creates the /var/netwitness/nw-backup directory (or the directory passed with the -b option), then using a combination of mongo and salt queries, will create the all-systems file in that directory. The all-systems file is used by the other scripts , with entries that contain the following in comma separated format:
DeviceType,Hostname,IPAddress,MinionID,SerialNumber
DeviceType = NRT Category Type (i.e. AdminServer,Broker,Concentrator,Decoder, etc.)
Hostname = Short hostname (not FQDN)
IPAddress = Management Interface IP address of host
MinionID = Unique Salt MinionID of host
SerialNumber = Device Serial Number for Reference and Support
Example:
AdminServer,nw-admin,192.168.1.129,70f95dc0-3cb6-4fd4-b9f2-ac923d0ba594,PK10T51
ESAPrimary,nw-esa,192.168.1.131,a598cb6b-4bd2-4ba2-af6a-79df3dab35e6,R9L8LNM
Broker+Search,nw-broker,192.168.1.130, 2a83b597-7970-4872-b76f-109cb591fa90,CSZ77X2
LogHybrid,nw-loghyb,192.168.1.133,87fc872c-68e3-45e3-9108-e30f847dc14e,PK10T0A
Malware,nw-malware,192.168.1.132,2c98e425-57a0-47d2-82d7-15795a6165f5,R90BCFWP
NetworkHybrid,nw-nethyb,192.168.1.134,9a99294e-3889-48b0-9555-11d3c21e2018,R90218K6
The script is designed to run from the cron on a regular basis (if you are in a dynamic environment were systems are added/removed on a regular basis), it has a 30 second timeout on the one question it asks.
If changes to the environment are detected, generation of new-systems and/or old-systems files will occur. The new-systems file can be used by other scripts for running specific targeted actions against the newly installed hosts. If a system is “offline” or has been removed from the UI, the old-systems file will have the entries for those hosts, so information about them is not lost.
./ssh-propagate.sh
Configuration options
Note: All command-line options are optional.
Run with no options, script uses the /var/netwitness/nw-backup/all-systems file to target
all hosts and copy the root users ecdsa-521 bit public ssh key to all hosts
Usage:
./ssh-propagate.sh [ -u ] [-c] [ -b ] [ -t ]
-b : path to the location of all-systems file. Default: (/var/netwitness/nw-backup)
-u : User to propagate keys to on all nodes (must exist on NW Server). Default: (root)
-g : Group associated with . Default: (username)
-c : Used with -u [ -g ] for non-root user, creates user on remote hosts.
-t : Target hosts (new-systems or anything grep-able from all-systems. Default: (ALL)
Run on the NW Admin Server. Performs the following (depending on options):
vi /root/scripts/nw-base.nrt
The default /etc/netwitness/recoverytool/nw-base.nrt file, distributed file with 11.x systems, only contains the following entries:
name nw-base
directory /etc/netwitness/platform/nodeinfo
file /etc/machine-id
# unmanaged files
stash /etc/fstab
stash /etc/hosts
stash /etc/sysconfig/iptables
stash /root/.ssh
# for azure
stash /etc/krb5.conf
stash /etc/logrotate.d/waagent.logrotate
stash /etc/mdadm.conf
stash /etc/waagent.conf
# only for nrt mode, run dnsclient to update the hosts configuration
after-import if [ "${NRT_MODE}" != 'standby' ]; then chef-client --config /var/lib/netwitness/config-management/client.rb --logfile /var/log/netwitness/config-management/chef-client.log --log_level info --runlist 'recipe[nw-dns-client]' > /dev/null 2>&1; fi
The “STASH” entries are NOT restored during an NRT import (recovery), but are available for reference in the /var/netwitness/backup/unmanaged folder, after the import. The included nw-base.nrt file has an expanded list of stash files and directories to include files used at several customer installations. To Fully restore functionality, these files need to be available after the restore.
Extended nw-base.nrt supplied with the scripts:
name nw-base
directory /etc/netwitness/platform/nodeinfo
file /etc/machine-id
# unmanaged files
stash /etc/fstab
stash /etc/hosts
stash /etc/resolv.conf
stash /etc/nsswitch.conf
stash /etc/passwd
stash /etc/shadow
stash /etc/group
stash /etc/sudo.conf
stash /etc/sudoers
stash /etc/sudoers.d
stash /etc/exports
stash /etc/krb5.conf
stash /etc/nfs.conf
stash /etc/ntp.conf
stash /etc/rsyslog.conf
stash /etc/logrotate.conf
stash /etc/sysconfig/network
stash /etc/sysconfig/nfs
stash /etc/sysconfig/iptables
stash /etc/sysconfig/iptables.bak
stash /etc/sysconfig/iptables-config
stash /etc/crontab
stash /etc/sysconfig/network-scripts/ifcfg-em1
stash /etc/sysconfig/network-scripts/ifcfg-em2
stash /etc/sysconfig/network-scripts/ifcfg-em3
stash /etc/sysconfig/network-scripts/ifcfg-em4
stash /etc/cron.hourly
stash /etc/cron.daily
stash /etc/cron.weekly
stash /etc/cron.daily
stash /etc/pam.d/netwitness
stash /etc/pam.d/securityanalytics
stash /etc/logrotate.d
stash /etc/logstash
stash /etc/multipath.conf
stash /etc/lvm
stash /etc/raddb
stash /etc/rsyslog.d
stash /etc/snmp
stash /etc/ssh
stash /var/ace
stash /home
stash /root (note: this backs up ALL file/folders in /root including /root/.ssh)
stash /var/netwitness/nw-backup/all-systems
# for azure
stash /etc/krb5.conf
stash /etc/logrotate.d/waagent.logrotate
stash /etc/mdadm.conf
stash /etc/waagent.conf
# only for nrt mode, run dnsclient to update the hosts configuration
after-import if [ "${NRT_MODE}" != 'standby' ]; then chef-client --config /var/lib/netwitness/config-management/client.rb --logfile /var/log/netwitness/config-management/chef-client.log --log_level info --runlist 'recipe[nw-dns-client]' > /dev/null 2>&1; fi
Edit the file to include any additional locations (files or directories) that contain customizations in your deployment, then save the file in the same directory as the nw-backup.sh script. The backup script will verify the file on each system matches your modified file and if not, will automatically copy the modified file to each host before running NRT on that host.
./nw-backup.sh
Configuration Options:
Note: All command-line options are optional.
With no options selected, the script backup ALL devices listed in the
/var/netwitness/nw-backup/all-systems file(except any commented(#) out)
and copy the backup files to the /var/netwitness/nw-backup/ / directory
on the NW Server.
Usage:
./nw-backup.sh [-b ] [-m ] [-l ] [-p ] [-s ] [-d ] [-t ] [-u ] [-g ] [-U ] [-G ] -M -L -R -I
Options:
-b : path for NRT backup files. Default: (/var/netwitness/backup)
-m : remote transfer mode (scp or nfs). Default: (scp)
-p : Path on NW server for logs and location of all-systems
file. Default: (/var/netwitnes/nw-backup)
-d : Path on destination server to move completed backup files to
via nfs or scp. Default: (/var/netwitness/nw-backup)
-s : Destination server IP address for storing completed backup files,
transferred via nfs or scp. Default: (NW Server IP)
-u : User acct for SCP transfers of completed backups, user must exist on all
target systems and on Destination server. Default: (root)
-g : Group for assigning permission to files copied to remote host. Default (root)
-U
: User acct on Remote system SCP transfers of completed backups, user must exist on
remote server adn have SSH-Key auth configured. Default: (root)
-G
: Group associated with Remote SCP user for transfer of completed backups, group must
exist on remote server. Default (root)
-l : local mount point for NFS share. Default: (/mnt/backup)
-t
: backup ONLY specific Target(s), (can be anything greppable from all-systems file)
Default: (all)
Special Targets:
core (Broker, Concentrator, Decoder, LogDecoder, Archiver,
LogCollector(vlc), NetworkHybrid, LogHybrid)
nonw (all devices except AdminServer)
nwonly (AdminServer only)
esaonly (all ESA devices only)
endpoint (all endpoint devices EndpointHybrid, EndpointLogHybrid, Gateway)
Exclusions/Inclusions/Service Control:
-M : Exclude the Malware Analysis File Store.
-I : Include the Broker Index files for RMA/Tech Refresh situation. Default: (Exclude)
-L : DO NOT STOP LogCollector (nwlogcollector) service during backup. Default: (Stop Service)
-R : DO NOT STOP Reporting Engine (rsasoc_re) service during backup. Default: (Stop Service)
Note: NRT normal operation stops these services, not stopping will affect:
LogCollector - Will not have latest tracking data for some logs.
Reporting Engine - Some live chart data and alert status data may be lost.