Project

General

Profile

Backup Setup » History » Version 6

Jon Goldberg, 11/09/2020 07:36 PM

1 6 Jon Goldberg
{{last_updated_at}} by {{last_updated_by}}
2
3 1 Jon Goldberg
# Backup Setup
4
5
## Overview
6
7
For a server to be backed up to Megaphone Tech's standards, all of the following must be true:
8
9
-   The backup must happen at least daily.
10
-   The backup must be tested (and testable) to ensure its validity.
11
-   The backup must be encrypted in transit and at rest. 
12
-   If the backup resides on a server outside of our control, the data must be encrypted such that those controlling the server can not read the data.
13
-   At least one copy of the backup must be in a separate geographical location from the original data.
14
-   Databases must be backed up using a database dump tool and stored in a backed-up area of the filesystem.
15
-   The backup should be monitored for both successes and failure.  Alerts should be generated for failed backups, and for backups that don't run.
16
17
To accomplish this, we use a modified copy of [backupninja](https://0xacab.org/riseuplabs/backupninja) to manage the backups.  It reports into our centralized [[Icinga2]] monitoring.
18
The preferred back-end for backups is [borgbackup](https://borgbackup.readthedocs.io/en/stable/), which provides for validity testing and client-side encryption.
19
20 5 Jon Goldberg
## Ansible
21
* Assign the server to a group with the `backupninja` role.
22 1 Jon Goldberg
23 5 Jon Goldberg
## Step-by-step (manual) guide
24 1 Jon Goldberg
25 5 Jon Goldberg
{{collapse
26
27 1 Jon Goldberg
**Note:** This entire guide assumes you're running as *root* on all
28
servers during setup.
29
30
### Icinga Server setup
31
32
-   Get the API User password for the "backupninja" user from `/etc/icinga2/conf.d/api-users.conf`.  This is `ICINGA2_API_PASSWORD`, below.
33
-   Edit the appropriate host conf file (in `/etc/icinga2/conf.d/hosts` to include the line:
34
35
```
36
has_backupninja: true
37
```
38
39
-   Also note the exact name of the `Host` object on line 1 of the file.  This is `ICINGA2_HOSTNAME` below.
40
-   Run `service icinga2 checkconfig && service icinga2 reload` for your change to take effect.
41
42
### Monitored Server setup
43
44
-   Install backupninja and borg.
45
46
```bash
47
#Ubuntu 16.04+
48
apt install backupninja borgbackup
49
#Debian Jessie
50
apt install backupninja
51
apt install -t jessie-backports borgbackup
52
#CentOS 7.3
53
#ensure epel repo is enabled
54
yum install backupninja borgbackup
55
```
56
57
-   If you're using MySQL 5.7+, you can't export the `information_schema` table.  There's a proposed patch for backupninja to exclude it, which you should apply:
58
59
```bash
60
cd /usr/share/backupninja/
61 4 Joseph Lacey
wget -O mysql.patch https://gist.githubusercontent.com/MegaphoneJon/94543829a2dfd6b3ed216b646afb0e8f/raw/abe58456b57eb123a1cf0023adb92ad2fa890cb1/backupninja%2520MySQL%25205.7%2520support
62 1 Jon Goldberg
#Ignore "patch unexpectedly ends in middle of line" warning
63
patch -p0 < mysql.patch
64
rm mysql.patch mysql.orig
65
```
66
67
-   Append this to the end of `/usr/sbin/backupninja`:
68 4 Joseph Lacey
    <https://gist.github.com/MegaphoneJon/322a4fea5707013433d9763972e4d414>
69 1 Jon Goldberg
-   Set up a local borg repo.
70
71
```bash
72
# Generate a password locally with a password generator like pwgen.
73
borg init /opt/borg
74
```
75
76
-   Set up a remote borg repo on rsync.net.
77
78
```bash
79
# Copy the root user's public key to rsync.net's authorized_keys
80
# If no key exists, create one with no passphrase
81
# Source: http://www.rsync.net/resources/howto/ssh_keys.html
82 2 Jon Goldberg
cat ~/.ssh/id_rsa.pub | ssh 8139@usw-s008.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
83 1 Jon Goldberg
84
# Generate a password locally with a password generator like pwgen.
85
# Replace "lava" with the name of the borg repo you'd like to create.
86 2 Jon Goldberg
borg init 8139@usw-s008.rsync.net:lava --remote-path=/usr/local/bin/borg1/borg1
87 1 Jon Goldberg
```
88
89
-   Put a set of standard configuration files in `/etc/backup.d`.
90
91
```bash
92
cd /etc/backup.d
93 4 Joseph Lacey
wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/10-info.sys
94
wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/30-databases.mysql
95
wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/50-borg-local.sh
96
wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/60-borg-remote.sh
97 1 Jon Goldberg
chmod 600 *
98
```
99
100
-   Edit the backupninja config(s) for borg to set the repository name and passphrase.
101 3 Jon Goldberg
    -   e.g. local repository is `/opt/borg` and remote repository is `8139@usw-s008.rsync.net:orange`
102 1 Jon Goldberg
-   Add to /etc/backupninja.conf:
103
104
```bash
105
ICINGA2_API_USER=backupninja
106
ICINGA2_SERVER_ADDRESS=orange.megaphonetech.com
107
ICINGA2_API_PORT=5665
108
ICINGA2_API_PASSWORD=<see above>
109
ICINGA2_HOSTNAME=<see above>
110
```
111
112
### Document
113
114
-   Update the [internal CRM server list](https://crm.megaphonetech.com/server-list) to reflect the correct backup method.
115
-   Record the borg passphrase(s) in the password manager.  This is very important; otherwise the backup is unrecoverable.
116 5 Jon Goldberg
}}