Migrating models

There has been some discussion on the app engine list recently about the best way to handle data migration when your model changes. I thought I'd put together a short summary of some of the options which are open to add, rename, remove or change the type of an existing field. For the examples here, I'm going to start with a simple model example and gradually change the class. At each stage we can see how the data store handles the change. >>> from google.appengine.ext import db >>> class Entity(db.Model): ... """Dummy class for testing""" ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... city = db.StringProperty() ... birth_year = db.IntegerProperty() ... height = db.IntegerProperty() We also want some test data. I think one name should be enough here. We'll add it to the data store and then read it back to check it's there: >>> e = Entity(first_name='Abraham', last_name='Lincoln', city='Washington', height=76, birth_year=1865).put() >>> e = Entity.all().filter('last_name =', 'Lincoln')[0] >>> print e.first_name, e.city Abraham Washington Adding a field Adding a new field is easy enough, we get it set to the default value (or None if there is no default): >>> class Entity(db.Model): ... """Dummy class for testing""" ... version = db.IntegerProperty(default=2) ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... city = db.StringProperty() ... birth_year = db.IntegerProperty() ... height = db.IntegerProperty() >>> e = Entity.all().filter('last_name =', 'Lincoln')[0] >>> print e.first_name, getattr(e, 'version', 'not set') Abraham 2 Watch out though, until you commit an entity the new field won't be accessible in queries: >>> print [e.first_name for e in Entity.all().filter('version =', 2)] [] >>> for e in Entity.all(): key = e.put() >>> print [e.first_name for e in Entity.all().filter('version =', 2)] [u'Abraham'] Renaming a field How about renaming a field? There is of course no relationship between the original and the new field value: >>> class Entity(db.Model): ... """Dummy class for testing""" ... version = db.IntegerProperty(default=2) ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... city = db.StringProperty() ... yob = db.IntegerProperty() ... height = db.IntegerProperty() >>> e = Entity.all().filter('last_name =', 'Lincoln')[0] >>> print e.first_name, e.yob Abraham None To rename a property (in this example birth_date becomes yob), create a property with the new name and make the original into a DummyProperty. Then a bit of fixup code in the class __init__ can handle the migration: >>> class DummyProperty(db.StringProperty): ... """A property whose only value is None and nothing ... gets saved to the data store""" ... def validate(self, value): ... return None ... def get_value_for_datastore(self, model_instance): ... return [] The actual renaming is done by picking up the old and new names as keyword arguments to the initialiser and the doing any required conversion: >>> class Entity(db.Model): ... """Dummy class for testing""" ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... city = db.StringProperty() ... birth_year = DummyProperty() # Old field ... yob = db.IntegerProperty() # New field ... height = db.IntegerProperty() ... def __init__(self, parent=None, key=None, _app=None, ... birth_year=None, yob=None, **kwds): ... super(Entity,self).__init__(parent, key, _app, ... yob = birth_year if birth_year is not None else yob, ... **kwds) >>> e = Entity.all().filter('last_name =', 'Lincoln')[0] >>> print e.first_name, e.yob, e.birth_year Abraham 1865 None To update the datastore just grab a bunch of objects which have the old field (sorting the query works as an existence test) and put each entity. Once we've done that the field disappears from the index in future queries, but it does become accessible using the new attribute name: >>> len([ e.put() for e in Entity.all().order('birth_year').fetch(10)]) 1 >>> len([ e.put() for e in Entity.all().order('birth_year').fetch(10)]) 0 >>> [ e.first_name for e in Entity.all().order('yob').fetch(10)] [u'Abraham'] Once the migration has been completed we can remove the DummyProperty from the model. If the old field isn't a property which supports sorting then you'll need to find another way to do the migration: using the version field would be good here, but only as long as you know the version number is actually set in all the data. Removing a field Removing a field is not a problem, the extra data in the database is simply ignored: >>> class Entity(db.Model): ... """Dummy class for testing""" ... version = db.IntegerProperty(default=2) ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... yob = db.IntegerProperty() ... height = db.IntegerProperty() >>> e = Entity.all().filter('last_name =', 'Lincoln')[0] >>> print e.first_name, getattr(e, 'city', 'not set') Abraham not set Alternatively we can make it into a dummy property and use the same trick as before to make sure the content is actually removed: >>> class Entity(db.Model): ... """Dummy class for testing""" ... version = db.IntegerProperty(default=2) ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... city = DummyProperty() ... yob = db.IntegerProperty() ... height = db.IntegerProperty() >>> len([ e.put() for e in Entity.all().order('city').fetch(10)]) 1 >>> len([ e.put() for e in Entity.all().order('city').fetch(10)]) 0 >>> e = Entity.all().filter('last_name =', 'Lincoln')[0] >>> print e.first_name, e.city Abraham None Changing field type If we need to change the type of a property and don't try to handle it, then we'll get an exception thrown >>> class Entity(db.Model): ... """Dummy class for testing""" ... version = db.IntegerProperty(default=2) ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... yob = db.IntegerProperty() ... height = db.FloatProperty() ... >>> [(e.first_name, e.height) for e in Entity.all()] Traceback (most recent call last): File "<stdin>", line 1, in <module> BadValueError: Property height must be a float Fortunately that can be fixed using __init__ in the same way as a rename. This time (for variety) I decided to use the version number to force the migration: >>> class Entity(db.Model): ... """Dummy class for testing""" ... version = db.IntegerProperty(default=3) ... first_name = db.StringProperty() ... last_name = db.StringProperty() ... yob = db.IntegerProperty() ... height = db.FloatProperty() ... def __init__(self, parent=None, key=None, _app=None, ... height=None, version=None, **kwds): ... if height is not None: height = float(height) ... super(Entity,self).__init__(parent, key, _app, ... height = height, version=3, ... **kwds) >>> len([ e.put() for e in Entity.all().filter('version <', 3).fetch(10)]) 1 >>> len([ e.put() for e in Entity.all().filter('version <', 3).fetch(10)]) 0 >>> [(e.first_name, e.height) for e in Entity.all()] [(u'Abraham', 76.0)] You can also use this to fixup bad content: if something gets into the datastore which won't validate then you can't read it, but if you can recognise the bad data you can check for it in __init__ and correct it.

Decoding a Query back to a string

On the App Engine Google group, Thomas Kuczek asked: I have a query object representing a query. How can I print the resulting Gql to log it with the logger framework? I thought this sounded like an interesting and possibly useful thing to do, so I wrote a small module which can convert either a db.Query or a db.GqlQuery object to a meaningful string. Warning This code depends on details of the query implementation (a lot of the required fields are private), so I can be fairly safe in saying that not only may it break in the next release of the App Engine environment, but it almost certainly will break. In the code samples which follow the test class is: class Person(db.Model): """Dummy class for testing""" first_name = db.StringProperty() last_name = db.StringProperty() city = db.StringProperty() birth_year = db.IntegerProperty() height = db.IntegerProperty() First some setup code for the tests: >>> from google.appengine.ext import db >>> from Person import Person >>> from showquery import showQuery, showGqlQuery Now try printing some queries: >>> q = Person.all() >>> print showQuery(q) Person.all() >>> print showQuery(q.filter("last_name =", "Smith")) Person.all().filter('last_name =', 'Smith') If we have more than one filter, showQuery will always output them in sorted order rather than the order in which they were input. This is simply to make doctests easier: >>> print showQuery(q.filter('height <', 72)) Person.all().filter('height <', 72).filter('last_name =', 'Smith') >>> print showQuery(q.order("-height")) Person.all().filter('height <', 72).filter('last_name =', 'Smith').order('-height') We can even handle an ancestor on the query although the key may be a bit of a mouthful: >>> p = Person(first_name='Duncan', last_name='Booth', city='Oxford', height=183) >>> key = p.put() >>> print showQuery(q.ancestor(p)) Person.all().ancestor(datastore_types.Key.from_path('Person', 1, _app=u'test_app')).filter('height <', 72).filter('last_name =', 'Smith').order('-height') There is also a showGqlQuery function to convert GQL back to the equivalent query: >>> q = db.GqlQuery("SELECT * FROM Person WHERE last_name = :1 AND height < :2", "Smith", 72) >>> print showGqlQuery(q) SELECT * FROM Person WHERE last_name = :1 AND height < :2 Notice that once again the output may have the clauses in a different order than they were originally input. >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE last_name = :name AND height < :height")) SELECT * FROM Person WHERE height < :height AND last_name = :name We can also handle literal values in queries: >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE last_name = 'Smith'")) SELECT * FROM Person WHERE last_name = 'Smith' Sorting is also handled. The ORDER BY clause does preserve the original order: >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE height < :1 ORDER BY last_name ASC")) SELECT * FROM Person WHERE height < :1 ORDER BY last_name ASC >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE height<:1 ORDER BY last_name DESC, height ASC")) SELECT * FROM Person WHERE height < :1 ORDER BY last_name DESC, height ASC Ancestor, limit, and offset classes also all work. If you specify limit and offset separately then they are output together: >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE ANCESTOR IS :1 AND height < 72")) SELECT * FROM Person WHERE ANCESTOR IS :1 AND height < 72 >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE ANCESTOR IS :1 LIMIT 10,5")) SELECT * FROM Person WHERE ANCESTOR IS :1 LIMIT 10,5 >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE ANCESTOR IS :1 OFFSET 3")) SELECT * FROM Person WHERE ANCESTOR IS :1 OFFSET 3 >>> print showGqlQuery(db.GqlQuery("SELECT * FROM Person WHERE ANCESTOR IS 'xxx' LIMIT 3 OFFSET 5")) SELECT * FROM Person WHERE ANCESTOR IS 'xxx' LIMIT 5,3 The source code (showquery.py) looks like this: from google.appengine.ext import db from google.appengine.api import datastore def showQuery(query): """Represent a query as a string""" kind = query._model_class.kind() ancestor = query._Query__ancestor filters = query._Query__query_set orderings = query._Query__orderings hint = None limit = None offset = None res = ["%s.all()" % kind] if ancestor is not None: res.append("ancestor(%r)" % ancestor) for k in sorted(filters): res.append("filter(%r, %r)" % (k, filters[k])) for p, o in orderings: if o==datastore.Query.DESCENDING: p = '-'+p res.append("order(%r)" % p) return '.'.join(res) def showGqlQuery(query): """Represent a GQL query as a string""" proto = query._proto_query kind = query._model_class.kind() filters = proto.filters() boundfilters = proto._GQL__bound_filters orderings = proto.orderings() hint = proto.hint() limit = proto.limit() offset = proto._GQL__offset select = "SELECT * FROM %s" % kind where = [] order = [] for k in sorted(filters): for clause in filters[k]: name, op = clause if name==-1: name = 'ANCESTOR' where.append("%s %s :%s" % (name, op.upper(), k)) for k in sorted(boundfilters): where.append("%s %r" % (k, boundfilters[k])) for p, o in orderings: order.append("%s %s" % (p, 'DESC' if o==datastore.Query.DESCENDING else 'ASC')) gql = select if where: gql += ' WHERE '+' AND '.join(where) if order: gql += ' ORDER BY ' + ', '.join(order) if limit != -1: if offset != -1: gql += ' LIMIT %s,%s' % (offset,limit) else: gql += ' LIMIT %s' % limit elif offset != -1: gql += ' OFFSET %s' % offset return gql

Unit testing with Google App Engine

The App Engine isn't too friendly for testing: you can't use interactive mode, and there's some setup needed to get a test harness. I've found that using doctest works quite nicely with the appengine, provided you get that initial setup out of the way. If your application interacts with the datastore, then you probably want to set up some objects before querying them and to my mind that sits better with the doctest model of a long chatty document describing a pseudo-interactive session than the xUnit style of separate tests each starting from a clean system doing something and making a single assertion about the resulting state. For a particular project you can update the test framework to perform common setup for the tests. The structure I've come up with sets up the environment to run tests under the appEngine framework. It looks for tests in two places: *.py files in the project directory, and tests/*.tests. For the Python files doctest will run tests from all docstrings found in the file (and you must have at least one docstring in every Python source file otherwise you'll get an error, but you don't have to actually have any tests in it). Each docstring forms a separate test. For the other files they are just plain text (or you can use reST) and the entire file is run as one test (so the state is preserved until the end of the file). Here's a sample doctest file: DateTime testing The appengine DateTimeProperty property type just looks like an ordinary datetime object when you use it. >>> from datetime import datetime, timedelta >>> from google.appengine.ext import db >>> class Test(db.Model): ... date = db.DateTimeProperty(auto_now_add=True) ... date2 = db.DateTimeProperty() >>> obj = Test(date2=datetime(2008,1,1)) The date attribute is set automatically to the current time, but we can use ellipsis to skip over the part that varies and check that it is very nearly the current time: >>> obj.date #doctest: +ELLIPSIS datetime.datetime(...) >>> obj.date - datetime.now() < timedelta(1) True For the other attribute we can test the extact value: >>> obj.date2 == datetime(2008,1,1) True >>> obj.date2.strftime("%a, %d %b %Y %H:%M:%S +0000") 'Tue, 01 Jan 2008 00:00:00 +0000'   Download an example appengine project with this test and execute tests/runtests.py to run it.