Using Google App Engine Backup and Restore

BANGALORE, INDIA: When I started developing the web site for the web conference on Google App Engine towards the end of last year, it dawned on me pretty quickly that one of my main development challenges would be to find a way to export all of the data we would be collecting in case something went wrong with the deployment datastore.

The difference between having a backup and not was the difference between braving a few hours of down-time and canceling the conference because we had lost all of our registration data. Since Google App Engine does not come with a data backup solution (Google is currently working on its own datastore import and export utility for large datasets which is currently on the roadmap for the first quarter of 2009), it was up to me to build my own.

This is what prompted me to write Google App Engine Backup and Restore (or Gaebar, for short).

While designing Gaebar, I knew that having backups alone would not be enough if there wasn't an easy way to restore those backups. The real value of a backup, after all, is in the restore.

How it works

Backing up an entire datastore is not a trivial task on Google App Engine since App Engine currently has just one modus operandi. App Engine supports a massively-scalable model based on requests and responses where each response must return within ten seconds. It doesn't currently have a mode of operation that isn't required to scale but which supports long-running processes.

The current workaround is to fake long-running processes by using a client-side engine and breaking up batch operations into bite-sized portions that work within the massively-scalable request-response model. Building administrative features in this way is somewhat akin to using a Ferrari to go from your bedroom to the bathroom: it's overkill and quite uncomfortable but can be made to work if you want it badly enough. This is the method that Gaebar uses to enable easy backups and restores on the Google App Engine platform.

Gaebar backs up your datastore, a few rows at a time, into Python functions and stores these functions in python modules called code shards that are less than 1MB in size each. To restore your datastore, Gaebar runs these functions, in successive requests, to recreate the models and entities in your datastore.

In other words, Gaebar backs up your datastore to Python code and then runs that code to restore it. And, it does all this while staying within the 10 second execution limit and the 1MB size limit on data structures that are currently in effect on the deployment environment.

Gaebar automates the backup and restore process as much as possible, going so far as to automatically download your backups from the deployment environment onto your local development server.

And you control the whole process via an easy-to-use, if pink, online interface.

Back me up, Scotty!

First off, understand that Gaebar only works with Django projects. So, if you're using webapp, I'm sorry to say that you're currently out of luck. (If you happen to port Gaebar to webapp, please do let me know.)

First things first, a patch

Before you can run Gaebar, you must patch your dev_appserver.py as per the instructions here: http://aralbalkan.com/1440 (and please star issue 616 if you'd like Google to fix this so we can remove this step).

This is required in order to override some of the local development server restrictions to allow the automatic download of backups. Gaebar will not work unless you implement this patch.

Quick start: Running the test suites

The easiest way to get started with Gaebar is to download one of the two test suite applications, Gaebar Gead or Gaebar App-Engine-Patch.

The test suite applications are built on the Google App Engine Helper and app-engine-patch skeletons, respectively, and come pre-configured with Gaebar. You can simply download one of the test apps and follow the instructions to populate a test datastore, run a backup/restore, and run the test suite to make sure that everything is working.

Starting a new Django application

For your own projects, the easiest way to get started with Gaebar is if you're building a new Django application. You can simply download the Gaebar Gaed Skeleton which contains a base Google App Engine Django install, including a zipped up Django 1.x and Gaebar.

Adding Gaebar to an existing Django project

If you already have a Django project, you need to add the Gaebar app to it. (Gaebar is just a regular Django app). An overview of how to do this is provided below but you should really read the readme.txt file that comes with Gaebar for full step-by-step instructions.

Start by downloading the Gaebar. You can download Gaebar Beta 3 as a zip from GitHub or, if you're a Gitball Wizard, you can use Git to add the Gaebar master branch to your project as a submodule.

Once you have downloaded Gaebar, install it into your Django project by following this general overview:

   1. Modify urls.py to add the URL mappings for Gaebar
   2. Add the URL mapping for the Gaebar static assets folder to app.yaml
   3. Add the index used by Gaebar to your index.yaml
   4. Modify your settings.py to add the Gaebar settings that specify, among others, the location of your local development server and the models that you want to back up.

(Again, for step-by-step instructions, see the readme.txt file that comes with Gaebar.)

Once you've installed Gaebar, you can test that everything is working by firing up your local development server and accessing Gaebar at the /gaebar URL. (e.g., http://localhost:8000/gaebar). You should see the Gaebar interface.

Click here for more...!

(The above article is taken from code.google.com under Creative commons licensing)