It’s time for another release. s3w is a client to access and synchronise with Amazon Web Services’ Simple Storage Service (Amazon S3). It supports both direct access to S3 operations (GET, PUT, LIST, COPY, …) and higher-level functionality like bucket-to-bucket copy and pushing local directory structures into buckets.
s3w 0.6.0 is now out. As well as the standard array of bug fixes, the major new feature in this release is “stacked push”. In a stacked push, another bucket (or prefix within a bucket) is used as a base, and any files that match the corresponding object in the stacked-upon location won’t be pushed again. Incremental backups are one use for that; I also use it as a quick way to publish slightly different versions of files. Multiple stacked locations can be used, but each new location in the stack adds another network round-trip to slow the process down, so you probably want to keep the number low. They’re searched left-to-right as on the command line, push –stacked base1 –stacked base2 … src/ dest/.
Pull has a corresponding stacked mode for reconstructing the directory tree, which reads in the same order. There’s also new copybucket –move support, which deletes the source keys after they’re copied, and which can be paired with the new copybucket –save-date, which preserves the original Last-Modified date in a special piece of metadata which push will use (incremental backups, again). Etags (file checksums) can be calculated to save redundant pushes as well, which is useful if you have severe clock skew or unreliable modification times.
There are also scattered bug fixes to problems that emerged during the development cycle, and some extra niceties like offering suggestions to mistyped commands or bucket names. The configuration file can customise more behaviour – per-bucket exclusions and custom short names for buckets. See s3w config –help for details of how to set these values – push.bucket.<dom>.excludes (list) and shortname.<shortname> (string bucket name).
s3w depends on Python 2.6 and Boto. There are build and quickstart instructions in the readme. Running just s3w with no arguments will give a list of commands and a brief summary, while comprehensive documentation for each is available with s3w <commandname> –help.
Development takes place in a Bazaar branch, currently hosted on Launchpad (not necessarily going to stay there). You can access it using bzr branch lp:s3w. Patches welcome! There are some usage examples on the homepage, as well as in the internal documentation.