User Tools

Site Tools


rsync

Introduction

There are many ways to clone a file system, but only few are as efficient as rsync. Part of every possible *nix derivate rsync is a the file based cloning tool for live file systems. One of the most popular use cases is parts of a server to a standby machine or a slave system. rsync is optimized to transfer as little data between the systems as possible to minimize bandwidth usage.

The efficiency comes from comparing the filesystems prior to syncing them. There are different ways of calculating the differences. The simplest one is comparing timestamps. More sophisticated (but slower) methods include calculating hash sums.

What can rsync do for me?

Below I will use concrete examples to show you what you can do with rsync.

Use case 1: backing up your photo library

Okay here you are with your invaluable collection of digital photos of your family and journeys and you most certainly do not want to lose the to a harddisk crash. You want to back them up as often as possible, but this takes an aweful lot of time because of the sheer amount of Megabytes that you hoarded over all those years. Don't worry, rsync is here to save the day.

  1. Find out where your photo library resides
  2. Get an external drive (USB, different Computer or NAS) with enough free space
  3. Use rsync to backup the data by hand once
  4. Use cron to automate your backup

Finding your photo library

Since most people use applications to manager their photo collections, first step is to find out where your photos reside. I use Apples Aperture and put my photo library on an external firewire harddisk.

So the path to my library is:

/Volumes/FreeAgentDrive/Aperture_Library_on_FreeAgentDrive.aplibrary

Determining the path to the remote filesystem

For the moment I assume a filesystem that is already mounted. In my case its a remote directory on an NAS. The path were I want to store my backup is:

/Volumes/sebastian/backup

Using rsync to clone a directory and its contents

Usecase 2: Backing up a complete Linux server

Preparations

The list of exclusions

You don't want to syncronize all of the files from the server, especially not the pseudo filesystems like proc and sys.

Create a file that contains a blacklist of files and directories that are excluded from the backup with the following contents:

/proc
/sys
/mnt
/tmp
*.bak
*~

I suggest naming this file <server name>.exclude

Put an ssh key onto the server

In our scenario rsync will use ssh to connect to the remote server. Do run rsync in non-interactive mode you need to put a public ssh key on the server that does not use a password. For details on how to do this see the ssh wiki page.

Initial backup

We'll do the initial backup by hand, so that we see if everything is set up correctly.

rsync -e 'ssh -l <user>@<host>' -avHz --stats --exclude-from=<path to exclude file> <user>@<host>:<source dir> <target dir>

Be patient, the initial backup will be slow.

rsync.txt · Last modified: 2015/05/17 17:44 (external edit)