Backup Setup » History » Revision 2

« Previous | Revision 2/6 (diff) | Next »
Jon Goldberg, 06/30/2017 05:07 PM

Backup Setup


For a server to be backed up to Megaphone Tech's standards, all of the following must be true:

  • The backup must happen at least daily.
  • The backup must be tested (and testable) to ensure its validity.
  • The backup must be encrypted in transit and at rest. 
  • If the backup resides on a server outside of our control, the data must be encrypted such that those controlling the server can not read the data.
  • At least one copy of the backup must be in a separate geographical location from the original data.
  • Databases must be backed up using a database dump tool and stored in a backed-up area of the filesystem.
  • The backup should be monitored for both successes and failure.  Alerts should be generated for failed backups, and for backups that don't run.

To accomplish this, we use a modified copy of backupninja to manage the backups.  It reports into our centralized Icinga2 monitoring.
The preferred back-end for backups is borgbackup, which provides for validity testing and client-side encryption.

Currently, setup is manual and complicated.  When ansible is deployed, we can automate these steps.

Step-by-step guide

Note: This entire guide assumes you're running as root on all
servers during setup.

Icinga Server setup

  • Get the API User password for the "backupninja" user from /etc/icinga2/conf.d/api-users.conf.  This is ICINGA2_API_PASSWORD, below.
  • Edit the appropriate host conf file (in /etc/icinga2/conf.d/hosts to include the line:
has_backupninja: true
  • Also note the exact name of the Host object on line 1 of the file.  This is ICINGA2_HOSTNAME below.
  • Run service icinga2 checkconfig && service icinga2 reload for your change to take effect.

Monitored Server setup

  • Install backupninja and borg.
#Ubuntu 16.04+
apt install backupninja borgbackup
#Debian Jessie
apt install backupninja
apt install -t jessie-backports borgbackup
#CentOS 7.3
#ensure epel repo is enabled
yum install backupninja borgbackup
  • If you're using MySQL 5.7+, you can't export the information_schema table.  There's a proposed patch for backupninja to exclude it, which you should apply:
cd /usr/share/backupninja/
wget -O mysql.patch
#Ignore "patch unexpectedly ends in middle of line" warning
patch -p0 < mysql.patch
rm mysql.patch mysql.orig
# Generate a password locally with a password generator like pwgen.
borg init /opt/borg
  • Set up a remote borg repo on
# Copy the root user's public key to's authorized_keys
# If no key exists, create one with no passphrase
# Source:
cat ~/.ssh/ | ssh 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

# Generate a password locally with a password generator like pwgen.
# Replace "lava" with the name of the borg repo you'd like to create.
borg init --remote-path=/usr/local/bin/borg1/borg1
  • Put a set of standard configuration files in /etc/backup.d.
cd /etc/backup.d
chmod 600 *
  • Edit the backupninja config(s) for borg to set the repository name and passphrase.
    • e.g. local repository is /opt/borg and remote repository is
  • Add to /etc/backupninja.conf:
ICINGA2_HOSTNAME=<see above>


  • Update the internal CRM server list to reflect the correct backup method.
  • Record the borg passphrase(s) in the password manager.  This is very important; otherwise the backup is unrecoverable.

Updated by Jon Goldberg about 7 years ago · 2 revisions