Backing up Edgerouter and EdgeOS Device Configurations Automatically via ssh

ssh image

If you are deploying Edgerouter devices such as an ER-X, ERPro or other edgeOS device such as switches. You can back it up from the GUI easily enough. You can also use UNMS to perform backups. But what if you need to back them up and you are not connected to a Ubiquiti controller?

EdgeOS Automatic Backup + Verification

So our goal here is to create an automated backup that pulls a configuration from a named edgeOS device, and stores it locally for us.

We want to take this one step further by adding a little verification. We want to know if the file has been changed since the last backup. This way we know there was a configuration change made, and we can possibly take action on it.

By the way, while this post will reference the EdgeOS as our target, it will also work with VyOS, that EdgeOS is based on, as well as any other popular switch or router that allows you to pull a configuration via the CLI. Your commands might be slightly different for other platforms but you should be able to put it together easily enough.

The CLI SSH DIFF

One of the most powerful tools in computing is by far the CLI or command line interface, and we will use it here to perform some tasks for us.

For starters we have already configured our EdgeOS device to allow us to SSH into the system using ssh-keys. If you have not configured your device to do so, you will be prompted for a password. 

We want to ssh into the system and run the command

show configuration commands

You will get a list of all the “commands” needed to rebuild the system with the same configuration.

That was easy huh? But wait, you need to constantly press the spacebar to page all the way to the end, not ideal. And this is only a display on the EdgeOS device not on our local storage system. Lets fix that. But first we need to be able to display the entire text with no breaks or pauses.

Because the edgerouter, and EdgeOS is based on VyOS. We can export the VyOS commands directly

/opt/vyatta/sbin/vyatta-config-gen-sets.pl

You will notice that this will dump the entire content to the terminal .. no paging. Perfect!

SSH

By knowing the magic command to display the config without having to page the results, we have the next step of our backup solved. 

ssh [email protected] "/opt/vyatta/sbin/vyatta-config-gen-sets.pl" > edgeOS-YOUR.EDGEROUTER-backup.cfg

Replace YOUR.EDGEROUTER with your own edgeOS device IP or hostname and you should end up with a file in your current directory called edgeOS-*-backup.cfg. This file has everything you need to perform a restore of your device.

We won’t get into  file naming and storage in this post, but you should name them in a way that you can identify which file is the most current vs the previous one.

DIFF

So how do you know if a change was made? The linux DIFF command is the magic here.

In our case I have 2 file

$ ls -l erpoe5.config*
-rw-r--r--  1 ben  staff  28341 Oct 12 11:49 erpoe5.config.text
-rw-r--r--  1 ben  staff  28341 Oct 12 12:21 erpoe5.config2.text

These files do no have any changes. But lets look at the time stamps. We can see that one was saved later then the other, and it has a slightly different name. That is not important here, only that we can tell which was the “older” file.  

$ diff erpoe5.config2.text erpoe5.config.text
$

Executing the diff command will return nothing, because there is no difference in the files. They are EXACTLY the same.

So lets make a quick alteration, and try this again.

$ diff erpoe5.config2.text erpoe5.config.text
1c1

$ diff erpoe5.config2.text erpoe5.config.text
1c1
< set firewall all-ping 'disable'
---
>  set firewall all-ping 'enable'

This time we got data returned. This means that “something” changed. And look at that it tells us the exact line that is different. The “<” shows the left files value, and the “>” shows the right files value.

So in our case erpoe5.config.2.text is the newer file. ANd that file contains the changes that were made. Diff allows us to see what exactly was changed. In this case, disable all-ping on the firewall.

Notifiying of changes.

If you are like me, you want to know when a config file has been changed unexpectedly. Maybe the file is corrupted. Maybe someone made a typo and maybe you just want to keep a record. Whatever your reason, if DIFF shows an output, you can use that to send you an alert to check into it.

If you use Zabbix, you can use the zabbix-sender script to send a value to your Zabbix server, to a related trap. You can then use zabbix to send off notifications, open tickets, send an SMS notification, #slack notice or a simple email. 

Putting it all together.

Bellow is a link to a shell script on github that we made available for you. There are a few changes to this script, such as the diff command was altered to always return a result, this was so that we could store values consistently in our zabbix instance. 

Another change was the very basic file rotation that we used. The script only keeps 2 copies, it is up to you the user to decide how you want to store your files. Perhaps you only keep a primary configuration and only save additional files if there is a difference. 

Download it here

!/bin/bash
#
Description:
Demo of how to download EdgeOS configuration and check if a change exists.
If a change exists send a notification to a Zabbix server.
#
****
This script is provided as a demo to show how to backup configurations automatically
You should not consider this a production script. If you are interested in a production
script please contact me @ support[at]voice1[dot]me or feel free to modify this to your
own needs.
#
Uncomment the dated-backup line if you want to save a local file with the date.

****
#
By: VOICE1, LLC
LICENSE: BSD
#
DEVICE=10.0.0.1
ZABBIX_SERVER=10.0.0.200
ZABBIX_HOST='edgeOS-device'
DATE=date +%Y-%m-%d:%H:%M:%S
This overrides any existing file by the same name.
ssh ubnt@$DEVICE "/opt/vyatta/sbin/vyatta-config-gen-sets.pl" > edgeOS-$DEVICE-backup.cfg
dated-backup;
Uncomment to save currently backed up file with a date suffix.
cp edgeOS-$DEVICE-backup.cfg edgeOS-$DEVICE-backup-$DATE.cfg
Basic File rotation.
if [ ! -f edgeOS-$DEVICE-backup-new.cfg ]; then
# There is no "new" file, our current one becomes the new file.
cp edgeOS-$DEVICE-backup.cfg edgeOS-$DEVICE-backup-new.cfg
else
# The "new" file exists, but its now the old file because we just downloaded
# the latest config. This overrides the old config. Make sure you have it backed up!
cp edgeOS-$DEVICE-backup-new.cfg edgeOS-$DEVICE-backup-old.cfg
fi
if [ ! -f edgeOS-$DEVICE-backup-old.cfg ]; then
# There is no "older file, so theres really nothing to compare to.
# cp edgeOS-$DEVICE-backup.cfg edgeOS-$DEVICE-backup-old.cfg
echo "No older config to compare to. Pull another copy first."
exit(0)
fi
Diff
CHANGES=$(diff -qs edgeOS-$DEVICE-backup-old.cfg edgeOS-$DEVICE-backup-new.cfg)
change-notification;
Check the $CHANGES for either 'identical' or 'differ'
And notify zabbix of a change.

You can use any form of notification you want here, this is only a sample.
if [[ $CHANGES = differ ]]; then
zabbix_sender -z $ZABBIX_SERVER -s $ZABBIX_HOST -k edgeOS.config -o 'differ'
else
# No Changes Comment out if you do not want to receive notices of the same file.
echo "No changes detected."
zabbix_sender -z $ZABBIX_SERVER -s $ZABBIX_HOST -k edgeOS.config -o 'identical'
fi
exit(0)

Conclusion

Backing up your device configuration is a critical point for disaster recovery. Doing so regularly is important. Automating the task allows you to receive notifications of failures, and to quickly identify what actually changed. This process isn’t exclusive to EdgeOS, VyOS or other appliances. But to almost any device or configuration where a small change can set you back hours trying to identify or resolve. 

Check out our post on backing up XenServers as a primer to keeping your VM machines safe as well.

Leave a Reply