December 20th, 2012

Glacier transition support added to ZS3

Amazon Glacier came out a few months ago. Glacier is a system for storing data very cheaply, only $0.01 per gigabyte per month. It also has a high level of redundancy and reliability. The drawback, compared to S3, is that you can't access it on demand. It was added as a new API, and I looked into writing a CL library for it, but never found the time.

A few days ago Amazon announced integration between S3 and Glacier. Objects in S3 can be automatically transitioned to Glacier storage after a period of time. It's very similar to the automatic object expiration support already present in the S3 API.

Since I already have an S3 library, I updated it to support the new Glacier features.

The lifecycle-rule function has been revamped to support a new action argument, :transition. Instead of deleting the object when the rule is applied, it is transitioned to Glacier storage.

New functions restore-object and object-restoration-status let you initiate a Glacier object restoration and query the status of that restoration.

Here's an example:

  (zs3:create-bucket "")
  (setf (zs3:bucket-lifecycle "") 
        (zs3:lifecycle-rule :action :transition 
                            :prefix "glacier/" 
                            :days 7))

Any object uploaded to the "" with a prefix matching "glacier/" will be automatically transitioned to Glacier storage after 7 days. You can also use a :days option of 0, which means "transition as soon as possible." I tried that, and "as soon as possible" can mean hours. Apparently the process that does the transition does not run all the time.

After the object has transitioned, you can't retrieve it with functions like get-file:

  (zs3:get-file "" "glacier/archive.tgz" "/tmp/archive.tgz")
  Error: "InvalidObjectState: The operation is not valid for the object's storage class"

The object has to be restored first. Restoring an object works like this:

  (zs3:restore-object "" "glacier/archive.tgz" :days 1)

The "days" argument refers to the number of days the object is available for download before it automatically reverts back to Glacier storage.

Restoring an object can take several hours. To check on object's restoration status, you can do this:

  (zs3:object-restoration-status "" "glacier/archive.tgz")
  => "ongoing-request=\"true\""

The next release of Quicklisp will have this ZS3 update. To get it early, you can check it out from github or download it from my website. Enjoy!