There has been some discussion on the app engine list recently about the best way to handle data migration when your model changes. I thought I'd put together a short summary of some of the options which are open to add, rename, remove or change the type of an existing field. For the examples here, I'm going to start with a simple model example and gradually change the class. At each stage we can see how the data store handles the change.
We also want some test data. I think one name should be enough here.
Adding a fieldAdding a new field is easy enough, we get it set to the default value
Watch out though, until you commit an entity the new field won't be
Renaming a fieldHow about renaming a field? There is of course no relationship between
To rename a property (in this example birth_date becomes yob), create
The actual renaming is done by picking up the old and new names as
To update the datastore just grab a bunch of objects which have the
Once the migration has been completed we can remove the DummyProperty If the old field isn't a property which supports sorting then you'll Removing a fieldRemoving a field is not a problem, the extra data in the database is
Alternatively we can make it into a dummy property and use the same
Changing field typeIf we need to change the type of a property and don't try to handle
Fortunately that can be fixed using __init__ in the same way as a
You can also use this to fixup bad content: if something gets into the |
Wednesday, 28 May 2008
Migrating models
Posted by Duncan Booth at 17:46 1 comments
Labels: Google App Engine
Friday, 16 May 2008
Blogging with docutils
I've always liked using reStructuredText to write documents, it seems to fit with my way of thinking, and it is really quick to produce quite a pretty but clean looking document. So far for blogging I've been using Windows Live Writer to compose my posts, and to get the formatting and layout I want I usually have to drop down to HTML view and hack it around a bit, but for this blog I'm using reST along with a slightly hacked front end so that it supports syntax highlighting (no way am I going to do multi-coloured Python code manually). I grabbed the latest docutils from subversion. There seem to be several alternative versions of syntax highlighting in the docutils sandbox (and elsewhere on the web) but not yet a standard one. Eventually I settled on rst2html-highlight.py in the sandbox. Sadly it didn't do exactly what I wanted, so I had to modify it a bit. With the original I can write something like: Now try printing some queries: Which gives me:
but I added in the ability to format an included code block: In the code samples which follow the test class is: which results in:
Ok, I like that: I just wrote 4 slightly different includes to get those examples and absolutely no copy/paste. The highlighting is done using pygments which has a long list of supported languages. It also claims to come with a directive for docutils, but so far as I can see that isn't included in the egg so for now I'll stick with the one in docutils sandbox (of course if I installed docutils from an egg then I wouldn't have that one either). Once I've formatted the document then I cut and paste into Windows Live Writer and upload it to the blog. I updated my blog skin to include the appropriate css in the skin so any changes to formatting will then apply consistently throughout the site. Eventually I might try to shortcut the cut/paste/upload step but for now it gives me an easy way to preview the post in the final skin. You can find the code for all of this at http://code.google.com/p/kupuguy/source/browse/trunk/appengine-doctests |
Posted by Duncan Booth at 14:01 0 comments
Wednesday, 14 May 2008
Decoding a Query back to a string
On the App Engine Google group, Thomas Kuczek asked: I have a query object representing a query. How can I print the resulting Gql to log it with the logger framework? I thought this sounded like an interesting and possibly useful thing to do, so I wrote a small module which can convert either a db.Query or a db.GqlQuery object to a meaningful string. Warning This code depends on details of the query implementation (a lot of the required fields are private), so I can be fairly safe in saying that not only may it break in the next release of the App Engine environment, but it almost certainly will break. In the code samples which follow the test class is: class Person(db.Model): First some setup code for the tests: >>> from google.appengine.ext import db Now try printing some queries: >>> q = Person.all() If we have more than one filter, showQuery will always output them in sorted order rather than the order in which they were input. This is simply to make doctests easier: >>> print showQuery(q.filter('height <', 72)) We can even handle an ancestor on the query although the key may be a bit of a mouthful: >>> p = Person(first_name='Duncan', last_name='Booth', city='Oxford', height=183) There is also a showGqlQuery function to convert GQL back to the equivalent query: >>> q = db.GqlQuery("SELECT * FROM Person WHERE last_name = :1 AND height < :2", "Smith", 72) Notice that once again the output may have the clauses in a different order than they were originally input. >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE last_name = :name AND height < :height")) We can also handle literal values in queries: >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE last_name = 'Smith'")) Sorting is also handled. The ORDER BY clause does preserve the original order: >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE height < :1 ORDER BY last_name ASC")) Ancestor, limit, and offset classes also all work. If you specify limit and offset separately then they are output together: >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE ANCESTOR IS :1 AND height < 72")) The source code (showquery.py) looks like this: from google.appengine.ext import db |
Posted by Duncan Booth at 20:21 2 comments
Labels: Google App Engine
Monday, 5 May 2008
Unit testing with Google App Engine
The App Engine isn't too friendly for testing: you can't use interactive mode, and there's some setup needed to get a test harness. I've found that using doctest works quite nicely with the appengine, provided you get that initial setup out of the way. If your application interacts with the datastore, then you probably want to set up some objects before querying them and to my mind that sits better with the doctest model of a long chatty document describing a pseudo-interactive session than the xUnit style of separate tests each starting from a clean system doing something and making a single assertion about the resulting state. For a particular project you can update the test framework to perform common setup for the tests. The structure I've come up with sets up the environment to run tests under the appEngine framework. It looks for tests in two places: *.py files in the project directory, and tests/*.tests. For the Python files doctest will run tests from all docstrings found in the file (and you must have at least one docstring in every Python source file otherwise you'll get an error, but you don't have to actually have any tests in it). Each docstring forms a separate test. For the other files they are just plain text (or you can use reST) and the entire file is run as one test (so the state is preserved until the end of the file). Here's a sample doctest file: DateTime testing
Download an example appengine project with this test and execute tests/runtests.py to run it. |
Posted by Duncan Booth at 10:39 4 comments
Labels: Google App Engine
Introduction
I created this blog some time ago to write occasional technical wurblings or brain-dumps. However I also started a food blog at the same time and this one just became a test ground for playing with blog skins and seeing how things worked. No I've finally decided to start using this blog as well. Don't expect regular or frequent posts, or any kind of consistency of topics. |
Posted by Duncan Booth at 09:13 0 comments