I run some server backups to Amazon Simple Storage Service (S3) using s3sync. We were previously using s3backer and rsync to maintain daily and weekly backups, but it works out too bandwidth-inefficient to operate that way – what was saved on storage went out on incoming bandwidth in the same day.
Amazon has a special incoming bandwidth rate of 3c/GB for this month, for some reason I don’t remember, so I moved the backups to s3sync while it was cheap. It’s a lot more cost-efficient this way, but we lost the daily and weekly backups (unless all the data gets transferred multiple times). What I wanted was a way to clone an entire bucket into another, but I couldn’t find any existing tools to do that.
I had been looking at the excellent Boto AWS library for other reasons already, so I put together a little script with it. I can run s3w copybucket bucket1 bucket2 to do a bucket-to-bucket copy of all the keys without transferring the data out and in again. It copies as efficiently as possible, only overwriting if the existing key content is different in the destination than the source, and using Amazon’s key copying so there’s no bandwidth used. This version also includes putstdin, getstdout, metadata, and list commands that I found useful for other parts of the backup system and debugging.
This is very early stages, there’s no installer in that, and it’s probably full of bugs. I’ve been running it on our backups for a while now without any issues though. I have a more advanced version locally with some more commands in it I’ll release if it goes anywhere.