How to have a rock solid Linux backup without pro budget V2
You can check my original article here: Backing up Linux web server live via SSH.
Since developing the first variant I've discovered several drawbacks to the method:
- Bandwidth
The entire backup is retransmitted each day causing huge bandwidth expenses. - CPU load
Archiving and compressing the entire backup is not such a small task. You can imagine that the server gets pretty loaded during the backup which causes websites to be less responsive for extended periods of time. - Duration of the backup
My old version of backup took up to 2 and a half hours to complete. This is a substantial amount of time.
All of these factors prompted me to experiment with rsync and attempt a solution that is incremental. The result is operational on my machine for several days and I'm very pleased with improvements in both speed and CPU load.
The solution that I've come up with is to simply rsync all the important data on the server to a folder on my home machine and then archive that folder. This leaves me with MySQL server that needs to be archived in a separate process. What I did with MySQL is to setup a master/slave replication setup.
In this setup my home machine runs a MySQL server that is a slave of my main MySQL server. This way all the data from the main server is almost immediately mirrored to the slave server. So archiving my database has been reduced to simply mysqldumping contents of the slave MySQL server on my home machine.
Not to delay you any further here is the script that I use to backup my server at the moment:
#!/usr/bin/env bash
LOG_FILE="/var/log/evorion-backup.log"
BACKUP_HOST=david.evorion.hr
echo "" >> $LOG_FILE
echo "Evorion backup utility started at: `date "+%F %T"`" >> $LOG_FILE
# Must be run as root user
if [ "$UID" -ne "0" ]
then
echo "[`date "+%F %T"`] Error: You must run this script as root!" >> $LOG_FILE
exit 67
fi
echo "[`date "+%F %T"`] User id check succesful" >> $LOG_FILE
# Compress directly into ssh connection
echo "[`date "+%F %T"`] Dumping and archiving started" >> $LOG_FILE
nice -n 19 dpkg -l | gzip | ssh -q vlatko@$BACKUP_HOST 'cat > ~/backup/archive/instPkgs-`date "+%F_%T"`.gz'
nice -n 19 rsync -rlzt --del /root vlatko@david.evorion.hr:~/backup/archive/
nice -n 19 rsync -rlzt --del /home vlatko@david.evorion.hr:~/backup/archive/
nice -n 19 rsync -rlzt --del /etc vlatko@david.evorion.hr:~/backup/archive/
nice -n 19 rsync -rlzt --del /usr/virtualweb vlatko@david.evorion.hr:~/backup/archive/
nice -n 19 rsync -rlzt --del /usr/local/mail vlatko@david.evorion.hr:~/backup/archive/
ssh -q vlatko@david.evorion.hr 'touch ~/backup/backupCompleted'
echo "[`date "+%F %T"`] Dumping and archiving completed" >> $LOG_FILE
This script is complemented with another script on my home machine that completes the backup process by archiving the mirrored folder and dumping the MySQL slave.
#!/bin/bash
if [ -f ~/backup/backupCompleted ]; then
rm ~/backup/backupCompleted
tar cfz ~/backup/archive_`date "+%F_%T"`.tgz -C / home/vlatko/backup/archive;
mysqldump -u root -pMYSQLPASS --all-databases | gzip > /home/vlatko/backup/dbs_`date "+%F_%T"`.gz
fi
A short examination of the new system will reveal several key factors.
- All the compression activities have been moved to the home machine.
- Only the modifications are transferred as opposed to transferring a full backup.
- The time it takes for the main server to do it's part is drastically reduced. It takes as little as 25-30 minutes for the entire process with the new method with much less load and bandwidth consumption.
There is another added benefit to this setup. When I do development I used to copy sites to my machine so that I can freely mess them up. With this setup all of the sites are already on the machine and MySQL server contains the exact replicas. All I need to do is to create copies of the stuff that is already on my home machine. A much easier thing to do as opposed to transferring everything from the production server.
Let me know what you guys think and let's see if we can make this into a V3 any time soon!
Add comment