Consolidating your backups and maximizing NRT (NetWitness Recovery Tool)

Document created by Thomas Jones

on Jul 9, 2019•Last modified by Thomas Jones

on Jan 7, 2021

Version 7Show Document

Like • Show 1 Like1
Comment • 6
View in full screen mode

Use this process if you would like full control of your backups, otherwise I advise you use the NRT Wrapper Method for an automated approach, - Centralized Backup & Restore of NetWitness Version 11.2+ (A Wrapper Script for NRT)

Changes are inevitable and no one knows when a restore is going to be needed. Today backup and restore processes are standard, required, and are part of nearly all basic deployment strategies. With that in mind, I was tasked/required to backup a environment of 32 hosts and running NRT manually was not a option, so I had to try to automate it.

Scenario -

Regular backups from all the netwitness hosts throughout the environment in the event of a problem, emergency, or a upgrade.

Problem -

NRT is a manual process out of the box and requires a significant amount of time to process (run the script and relocate the backup files).

Reference:

https://community.rsa.com/docs/DOC-96912

Resolution -

Through some trial and error I built a script to automate the process that can be controlled with a cron job. The issue was not NRT per se, it was pulling the backup files to a central location. I initially thought chef may be able to pull these items back but unfortunately, pulling was not something Chef did well. I did experiment with some Chef solutions but none worked the way I wanted them to. So, I went back to old school private and public keys (because of scp interactive) to move the backups to secondary ESA - for storage only. The next issue I came across with NRT was that it was device specific (--category Decoder). This was extremely problematic due to the automation.

Solution -

Get the keys situated (interactive responses make this step necessary)
1. On the backup host (where all the backups will reside)
  1. cd ~/.ssh
  2. ssh-keygen -t rsa -b 2048 -N "" -f ~/.ssh/nwbackup_key
  3. Copy the public key file to the SA Server
2. On the SA Server
  1. Put the public nwbackup_key.pub in /tmp
  2. salt '*' cmd.run 'mkdir ~/.ssh' (creates the ssh directory on all netwitness hosts)
  3. salt '*' cmd.run 'chmod 700 ~/.ssh' (changes the permission of the ssh directory on all the netwitness hosts)
  4. salt-cp '*' file.copy /tmp/nwbackup_key.pub /tmp/nwbackup_key.pub
  5. salt '*' cmd.run 'chmod 755 /tmp/nwbackup_key.pub'
  6. salt '*' cmd.run 'cat /tmp/nwbackup_key.pub >> ~/.ssh/authorized_keys'
  7. salt '*' cmd.run 'rm -f /tmp/nwbackup_key.pub'
  8. Check to make sure the public key has been saved in the authorized_keys file on the hosts
3. Backup server
  1. Test the key by ssh root@host from the backup host server to one of the servers being backed up.
4. Command for SA Server (Backup)
  nw-recovery-tool --export --dump-dir /var/netwitness/nw-recovery-tool_backup --category AdminServer
5. Commands for (Backup)
  
  nw-recovery-tool --export --dump-dir /var/netwitness/nw-recovery-tool_backup --category NetworkHybrid
  nw-recovery-tool --export --dump-dir /var/netwitness/nw-recovery-tool_backup --category Decoder
  nw-recovery-tool --export --dump-dir /var/netwitness/nw-recovery-tool_backup --category Concentrator
  nw-recovery-tool --export --dump-dir /var/netwitness/nw-recovery-tool_backup --category Broker
6. Other host types: Archiver, Broker, Concentrator, Decoder, EndpointHybrid, EndpointLogHybrid, ESAPrimary, ESASecondary, LogCollector, LogDecoder, LogHybrid, Malware, NetworkHybrid, UEBA
7. Create a script for each host type in your environment
  1. Put the scripts on the SA Server and distribute to the environment using salt-cp '*' file.copy command.
    1. Example script for a broker
    3. Notice the --category this is what will change from script to script. The zip file created contains the host name and date.
8. Create a script for the backup server (in my case it is a Secondary ESA)
  1. Example of the manual backup
    2. Iterate through the hosts using your script.

Summary:

Generated public/private key for the backup host
Distributed the public key via salt on the SA Server
Validated shh non-interactive connectivity between the hosts from the backup server
Created a backup script for each host type (broker, decoder, etc.)
Distributed the host scripts via salt on the SA Server
Created a script on the backup server to run the host scripts and then scp them back to the backup server.

In the end, my solution ended up having 8 scripts (same script with edits to the device type by services and category). The goal was to put all the backup files into one place so I could copy them and put them in a safe place. Please keep in mind, this is just one way of many possibilities to centralize backups and it is a solution I chose based on the environment I was working on.

Let me know your thoughts.

Tom J

Attachments

Outcomes

Tamás Szilágyi Jul 15, 2019 10:49 AM

This is great, it has some really good ideas - I've never tried to use chef to distribute keys to the other hosts.

The next issue I came across with NRT was that it was device specific (--category Decoder). This was extremely problematic due to the automation.

Not necessarily, you can get that information from the Admin servers mongo database using this:

mongo admin -u deploy_admin -p <deployPW> --eval "db=db.getSiblingDB('orchestration-server');db.getCollection('host').find({}, { _id: 0, _class: 0, version: 0, thirdParty: 0, meta: 0}).forEach(function(f){print(tojson(f, '', true));})"

-where the <deployPW> is your deployment password. This will generate an output like this:

{ "hostname" : "192.168.1.1", "displayName" : "NW11-2-SA", "installedServices" : [ "AdminServer" ] }

{ "hostname" : "192.168.1.2", "displayName" : "NW11-2-ESA", "installedServices" : [ "ESAPrimary" ] }
{ "hostname" : "192.168.1.3", "displayName" : "NW11-2-LDEC", "installedServices" : [ "LogDecoder" ] }
{ "hostname" : "192.168.1.4", "displayName" : "NW11-2-LCON", "installedServices" : [ "Concentrator" ] }

From this point, you can use 'cut' to get the last field (the service) for each host, and use it dynamically when making the commands for the hosts.

One more idea worth considering:

A CEF output can be added in case a backup failed, that way you can create alerts with mail notifications. Just send a log using the proper format to your designated logdecoder:

Of course you should customize this line to represent your deployment (version number, service name, log message, etc.). Also, firewall rules might be required between components.

One question: how did you backup services with deploy passwords (Adminserver, ESA)? For me this seemed to be the biggest issue, because I have to hardcode passwords in the scripts somehow.

1 person found this helpful

Actions

Thomas Jones

@ Tamás Szilágyi on Jul 15, 2019 1:53 PM

Hello Tamas,

I have the same problems. I have to back them up manually... the issue I have experienced is intensified 2 factor authentication. At least I can automate a large portion of the environment.

Also, I did not want to use your method above for hosts because of the deployadmin pw. But yes, that is a great observation.

Hopefully engineering will tie this all together to make it more realistic for large deployments.

Thanks for the feed back.

Tom J

Actions

Josh Randall

@ Thomas Jones on Jul 15, 2019 2:36 PM

The deploy_admin password can be variable-ized in scripts so that you do not need to hardcode the password or worry about it potentially changing, like so:

DEPLOY_PW=$(security-cli-client --get-config-prop --prop-hierarchy nw.security-client --prop-name platform.deployment.password --quiet)

There are multiple examples in scripts and blogs. Here are a few of them:

2 people found this helpful

Actions

Albert Cieszkowski Jan 6, 2021 5:19 AM

For anybody stumbling upon this topic - extended version of NRT automation is available:Centralized Backup & Restore of NetWitness Version 11.2+ (A Wrapper Script for NRT)

Actions

Thomas Jones

@ Albert Cieszkowski on Jan 6, 2021 11:34 PM

Thanks Albert,

I have added a note to the top of the post. This method is used as a one time setup and can be a little more difficult to deploy. The good news is after it is built, it is done and not something you have to mess with each upgrade.

Actions

John Snider

Jan 7, 2021 2:02 PM

While my centralized backup does occasionally need to be updated after an upgrade (last change was due to version upgrades in salt and mongodb that affected the output of the get-all-systems11.sh file) it is based on the same backup format and methods we have used since 10.5 (I wrote the backup scripts used for the 11.x migration, so created these to follow the same basic steps for familiararity) I plan on working with the dev team to integrate the functions into the base nw-recovery-tool script to expand it's capabilities, any input for changes or features are welcome.

One other caveat, is that with 11.5 if you add services to another host, say you add the "New Health and Wellness" to a broker or other server, using the mongo pull to get Category will give "Search, Broker" (Search is the New H&W), now with Tom's setup, you could just add a " -C Search as a second category (you can list multiples that way), but I'm working on a new version of the nw-backup scripts that will Auto Generate all components installed dynamically and create a custom "category.sequence" file for each host, this is something we have known would be an issue when we enable the ability to move different components around to different hosts. I'll post a new article when the new version is ready.

Actions

6 Comments

Tamás Szilágyi Jul 15, 2019 10:49 AM
This is great, it has some really good ideas - I've never tried to use chef to distribute keys to the other hosts.

The next issue I came across with NRT was that it was device specific (--category Decoder). This was extremely problematic due to the automation.
Not necessarily, you can get that information from the Admin servers mongo database using this:

mongo admin -u deploy_admin -p <deployPW> --eval "db=db.getSiblingDB('orchestration-server');db.getCollection('host').find({}, { _id: 0, _class: 0, version: 0, thirdParty: 0, meta: 0}).forEach(function(f){print(tojson(f, '', true));})"

-where the <deployPW> is your deployment password. This will generate an output like this:

{ "hostname" : "192.168.1.1", "displayName" : "NW11-2-SA", "installedServices" : [ "AdminServer" ] }
{ "hostname" : "192.168.1.2", "displayName" : "NW11-2-ESA", "installedServices" : [ "ESAPrimary" ] }
{ "hostname" : "192.168.1.3", "displayName" : "NW11-2-LDEC", "installedServices" : [ "LogDecoder" ] }
{ "hostname" : "192.168.1.4", "displayName" : "NW11-2-LCON", "installedServices" : [ "Concentrator" ] }

From this point, you can use 'cut' to get the last field (the service) for each host, and use it dynamically when making the commands for the hosts.

One more idea worth considering:
A CEF output can be added in case a backup failed, that way you can create alerts with mail notifications. Just send a log using the proper format to your designated logdecoder:

logger -n <LOGDECODER_IP> -P 514 -t "$(hostname)" "CEF:0|RSA|NetWitness Audit|11.2.0.0|BACKUP_FALIURE|Failed to backup component, check the logs|6|rt=$(date "+%b %d %Y %H:%M:%S") suser=admin sourceServiceName=SA_SERVER deviceProcessName=SA_SERVER outcome=Faliure"

Of course you should customize this line to represent your deployment (version number, service name, log message, etc.). Also, firewall rules might be required between components.

One question: how did you backup services with deploy passwords (Adminserver, ESA)? For me this seemed to be the biggest issue, because I have to hardcode passwords in the scripts somehow.
1 person found this helpful
Like • Show 1 Like1
Actions
- Thomas Jones @ Tamás Szilágyi on Jul 15, 2019 1:53 PM
  Hello Tamas,
  I have the same problems. I have to back them up manually... the issue I have experienced is intensified 2 factor authentication. At least I can automate a large portion of the environment.
  
  Also, I did not want to use your method above for hosts because of the deployadmin pw. But yes, that is a great observation.
  
  Hopefully engineering will tie this all together to make it more realistic for large deployments.
  
  Thanks for the feed back.
  
  Tom J
  Like • Show 0 Likes0
  Actions
  - Josh Randall @ Thomas Jones on Jul 15, 2019 2:36 PM
    The deploy_admin password can be variable-ized in scripts so that you do not need to hardcode the password or worry about it potentially changing, like so:
    
    DEPLOY_PW=$(security-cli-client --get-config-prop --prop-hierarchy nw.security-client --prop-name platform.deployment.password --quiet)
    
    There are multiple examples in scripts and blogs. Here are a few of them:
    https://community.rsa.com/community/products/netwitness/blog/2019/01/24/netwitness-storage-retention
    https://community.rsa.com/community/products/netwitness/blog/2019/04/29/examining-threat-aware-authentication
    https://community.rsa.com/community/products/netwitness/blog/2019/05/03/customizing-respond-incident-notification-emails
    https://community.rsa.com/community/products/netwitness/blog/2019/04/21/understanding-and-creating-endpoint-alerts-in-113
    2 people found this helpful
    Like • Show 2 Likes2
    Actions
Albert Cieszkowski Jan 6, 2021 5:19 AM
For anybody stumbling upon this topic - extended version of NRT automation is available:Centralized Backup & Restore of NetWitness Version 11.2+ (A Wrapper Script for NRT)
Like • Show 0 Likes0
Actions
- Thomas Jones @ Albert Cieszkowski on Jan 6, 2021 11:34 PM
  Thanks Albert,
  I have added a note to the top of the post. This method is used as a one time setup and can be a little more difficult to deploy. The good news is after it is built, it is done and not something you have to mess with each upgrade.
  Like • Show 0 Likes0
  Actions
John Snider Jan 7, 2021 2:02 PM
While my centralized backup does occasionally need to be updated after an upgrade (last change was due to version upgrades in salt and mongodb that affected the output of the get-all-systems11.sh file) it is based on the same backup format and methods we have used since 10.5 (I wrote the backup scripts used for the 11.x migration, so created these to follow the same basic steps for familiararity) I plan on working with the dev team to integrate the functions into the base nw-recovery-tool script to expand it's capabilities, any input for changes or features are welcome.

One other caveat, is that with 11.5 if you add services to another host, say you add the "New Health and Wellness" to a broker or other server, using the mongo pull to get Category will give "Search, Broker" (Search is the New H&W), now with Tom's setup, you could just add a " -C Search as a second category (you can list multiples that way), but I'm working on a new version of the nw-backup scripts that will Auto Generate all components installed dynamically and create a custom "category.sequence" file for each host, this is something we have known would be an issue when we enable the ability to move different components around to different hosts. I'll post a new article when the new version is ready.
Like • Show 0 Likes0
Actions

Consolidating your backups and maximizing NRT (NetWitness Recovery Tool)

Attachments

Outcomes

Related Content

Recommended Content