Project

General

Profile

Backup Setup » History » Revision 5

Revision 4 (Joseph Lacey, 12/16/2017 05:52 PM) → Revision 5/6 (Jon Goldberg, 03/01/2018 08:20 PM)

# Backup Setup 

 ## Overview 

 For a server to be backed up to Megaphone Tech's standards, all of the following must be true: 

 -     The backup must happen at least daily. 
 -     The backup must be tested (and testable) to ensure its validity. 
 -     The backup must be encrypted in transit and at rest.  
 -     If the backup resides on a server outside of our control, the data must be encrypted such that those controlling the server can not read the data. 
 -     At least one copy of the backup must be in a separate geographical location from the original data. 
 -     Databases must be backed up using a database dump tool and stored in a backed-up area of the filesystem. 
 -     The backup should be monitored for both successes and failure.  Alerts should be generated for failed backups, and for backups that don't run. 

 To accomplish this, we use a modified copy of [backupninja](https://0xacab.org/riseuplabs/backupninja) to manage the backups.  It reports into our centralized [[Icinga2]] monitoring. 
 The preferred back-end for backups is [borgbackup](https://borgbackup.readthedocs.io/en/stable/), which provides for validity testing and client-side encryption. 

 ## Ansible 
 * Assign the server to a group with the `backupninja` role. Currently, setup is manual and complicated.  When [ansible](https://www.ansible.com/) is deployed, we can automate these steps. 

 ## Step-by-step (manual) guide 

 {{collapse 

 **Note:** This entire guide assumes you're running as *root* on all 
 servers during setup. 

 ### Icinga Server setup 

 -     Get the API User password for the "backupninja" user from `/etc/icinga2/conf.d/api-users.conf`.  This is `ICINGA2_API_PASSWORD`, below. 
 -     Edit the appropriate host conf file (in `/etc/icinga2/conf.d/hosts` to include the line: 

 ``` 
 has_backupninja: true 
 ``` 

 -     Also note the exact name of the `Host` object on line 1 of the file.  This is `ICINGA2_HOSTNAME` below. 
 -     Run `service icinga2 checkconfig && service icinga2 reload` for your change to take effect. 

 ### Monitored Server setup 

 -     Install backupninja and borg. 

 ```bash 
 #Ubuntu 16.04+ 
 apt install backupninja borgbackup 
 #Debian Jessie 
 apt install backupninja 
 apt install -t jessie-backports borgbackup 
 #CentOS 7.3 
 #ensure epel repo is enabled 
 yum install backupninja borgbackup 
 ``` 

 -     If you're using MySQL 5.7+, you can't export the `information_schema` table.  There's a proposed patch for backupninja to exclude it, which you should apply: 

 ```bash 
 cd /usr/share/backupninja/ 
 wget -O mysql.patch https://gist.githubusercontent.com/MegaphoneJon/94543829a2dfd6b3ed216b646afb0e8f/raw/abe58456b57eb123a1cf0023adb92ad2fa890cb1/backupninja%2520MySQL%25205.7%2520support 
 #Ignore "patch unexpectedly ends in middle of line" warning 
 patch -p0 < mysql.patch 
 rm mysql.patch mysql.orig 
 ``` 

 -     Append this to the end of `/usr/sbin/backupninja`: 
     <https://gist.github.com/MegaphoneJon/322a4fea5707013433d9763972e4d414> 
 -     Set up a local borg repo. 

 ```bash 
 # Generate a password locally with a password generator like pwgen. 
 borg init /opt/borg 
 ``` 

 -     Set up a remote borg repo on rsync.net. 

 ```bash 
 # Copy the root user's public key to rsync.net's authorized_keys 
 # If no key exists, create one with no passphrase 
 # Source: http://www.rsync.net/resources/howto/ssh_keys.html 
 cat ~/.ssh/id_rsa.pub | ssh 8139@usw-s008.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc' 

 # Generate a password locally with a password generator like pwgen. 
 # Replace "lava" with the name of the borg repo you'd like to create. 
 borg init 8139@usw-s008.rsync.net:lava --remote-path=/usr/local/bin/borg1/borg1 
 ``` 

 -     Put a set of standard configuration files in `/etc/backup.d`. 

 ```bash 
 cd /etc/backup.d 
 wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/10-info.sys 
 wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/30-databases.mysql 
 wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/50-borg-local.sh 
 wget https://raw.githubusercontent.com/MegaphoneJon/backupninja_configs/master/60-borg-remote.sh 
 chmod 600 * 
 ``` 

 -     Edit the backupninja config(s) for borg to set the repository name and passphrase. 
     -     e.g. local repository is `/opt/borg` and remote repository is `8139@usw-s008.rsync.net:orange` 
 -     Add to /etc/backupninja.conf: 

 ```bash 
 ICINGA2_API_USER=backupninja 
 ICINGA2_SERVER_ADDRESS=orange.megaphonetech.com 
 ICINGA2_API_PORT=5665 
 ICINGA2_API_PASSWORD=<see above> 
 ICINGA2_HOSTNAME=<see above> 
 ``` 

 ### Document 

 -     Update the [internal CRM server list](https://crm.megaphonetech.com/server-list) to reflect the correct backup method. 
 -     Record the borg passphrase(s) in the password manager.  This is very important; otherwise the backup is unrecoverable. 
 }}