Megaphone Commons

# Backup Setup

## Overview

For a server to be backed up to Megaphone Tech's standards, all of the following must be true:

-   The backup must happen at least daily.

-   The backup must be tested (and testable) to ensure its validity.

-   The backup must be encrypted in transit and at rest. 

-   If the backup resides on a server outside of our control, the data must be encrypted such that those controlling the server can not read the data.

-   At least one copy of the backup must be in a separate geographical location from the original data.

-   Databases must be backed up using a database dump tool and stored in a backed-up area of the filesystem.

-   The backup should be monitored for both successes and failure.  Alerts should be generated for failed backups, and for backups that don't run.

To accomplish this, we use a modified copy of [backupninja](https://0xacab.org/riseuplabs/backupninja) to manage the backups.  It reports into our centralized [[Icinga2]] monitoring.

The preferred back-end for backups is [borgbackup](https://borgbackup.readthedocs.io/en/stable/), which provides for validity testing and client-side encryption.

Currently, setup is manual and complicated.  When [ansible](https://www.ansible.com/) is deployed, we can automate these steps.

## Step-by-step guide

**Note:** This entire guide assumes you're running as *root* on all

servers during setup.

### Icinga Server setup

-   Get the API User password for the "backupninja" user from `/etc/icinga2/conf.d/api-users.conf`.  This is `ICINGA2_API_PASSWORD`, below.

-   Edit the appropriate host conf file (in `/etc/icinga2/conf.d/hosts` to include the line:

```

has_backupninja: true

```

-   Also note the exact name of the `Host` object on line 1 of the file.  This is `ICINGA2_HOSTNAME` below.

-   Run `service icinga2 checkconfig && service icinga2 reload` for your change to take effect.

### Monitored Server setup

-   Install backupninja and borg.

```bash

#Ubuntu 16.04+

apt install backupninja borgbackup

#Debian Jessie

apt install backupninja

apt install -t jessie-backports borgbackup

#CentOS 7.3

#ensure epel repo is enabled

yum install backupninja borgbackup

```

-   If you're using MySQL 5.7+, you can't export the `information_schema` table.  There's a proposed patch for backupninja to exclude it, which you should apply:

```bash

cd /usr/share/backupninja/

wget -O mysql.patch https://gist.githubusercontent.com/PalanteJon/94543829a2dfd6b3ed216b646afb0e8f/raw/abe58456b57eb123a1cf0023adb92ad2fa890cb1/backupninja%2520MySQL%25205.7%2520support

#Ignore "patch unexpectedly ends in middle of line" warning

patch -p0 < mysql.patch

rm mysql.patch mysql.orig

```

-   Append this to the end of `/usr/sbin/backupninja`:

    <https://gist.github.com/PalanteJon/322a4fea5707013433d9763972e4d414>

-   Set up a local borg repo.

```bash

# Generate a password locally with a password generator like pwgen.

borg init /opt/borg

```

-   Set up a remote borg repo on rsync.net.

```bash

# Copy the root user's public key to rsync.net's authorized_keys

# If no key exists, create one with no passphrase

# Source: http://www.rsync.net/resources/howto/ssh_keys.html

cat ~/.ssh/id_rsa.pub | ssh 8139@usw-s008.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

# Generate a password locally with a password generator like pwgen.

# Replace "lava" with the name of the borg repo you'd like to create.

borg init 8139@usw-s008.rsync.net:lava --remote-path=/usr/local/bin/borg1/borg1

```

-   Put a set of standard configuration files in `/etc/backup.d`.

```bash

cd /etc/backup.d

wget https://raw.githubusercontent.com/PalanteJon/backupninja_configs/master/10-info.sys

wget https://raw.githubusercontent.com/PalanteJon/backupninja_configs/master/30-databases.mysql

wget https://raw.githubusercontent.com/PalanteJon/backupninja_configs/master/50-borg-local.sh

wget https://raw.githubusercontent.com/PalanteJon/backupninja_configs/master/60-borg-remote.sh

chmod 600 *

```

-   Edit the backupninja config(s) for borg to set the repository name and passphrase.

    -   e.g. local repository is `/opt/borg` and remote repository is `8139@usw-s008.rsync.net:orange`

-   Add to /etc/backupninja.conf:

```bash

ICINGA2_API_USER=backupninja

ICINGA2_SERVER_ADDRESS=orange.megaphonetech.com

ICINGA2_API_PORT=5665

ICINGA2_API_PASSWORD=<see above>

ICINGA2_HOSTNAME=<see above>

```

### Document

-   Update the [internal CRM server list](https://crm.megaphonetech.com/server-list) to reflect the correct backup method.

-   Record the borg passphrase(s) in the password manager.  This is very important; otherwise the backup is unrecoverable.