Sunday, April 26, 2009

Backup Rotations with Amazon S3

I've starting putting my backups onto Amazon S3 and I've already got a script working that does a nightly sync (let me know if you're interested in that and I'll do a blog post about it).

The next problem though is that I also want weekly snapshots on a one month rotation schedule, just in case it's a little while before I realise that I want a certain file back.

Option A: Setup some more cron jobs that will sync weekly to different S3 folders.
The problem with this is that I would be transferring the same files twice - once for the nightly sync and once for the weekly sync.

Option B: Take a snapshot of the nightly copy and put that in another folder.
This sounds better, and has the added benefit that I'm working with files that are already off my server, so won't be adding any extra load to it. The latest version of s3cmd supports copying between buckets and within buckets, but unfortunately does not (yet) support recursion in these scenarios. So my little challenge is to implement the recursion part with a shell script.

Now I'm no shell guru so improvement suggestions are most welcome:
I created a file called s3remotecp and put this into it:

#!/bin/sh

# check that two arguments have been supplied
if [ $# -ne 2 ]; then
echo 1>&2 Usage: $0 s3://sourcebucket/sourcefolder/ s3://destinationbucket/destinationfolder/
exit 127
fi

sourcePath=$1
destinationPath=$2

logfile='/var/log/s3remotecp.log'

for line in `s3cmd ls -r $sourcePath | awk '{print $4}'`;do
s3cmd cp $line $destinationPath`echo $line | awk -F "$sourcePath" '{print $2}'`
done


As of writing this, s3cmd is at version 0.9.9. The next version is likely to support recursion, which would make my little script redundant.

1 comment:

  1. s3cmd version 0.9.9.91 has now been released and supports recursion for cp and mv so this script isn't needed any more.

    just use s3cmd cp -r source destination

    ReplyDelete