A Primer on Python Metaclass Programming

A Primer on Python Metaclass Programmingby David
Mertz
04/17/2003
This series of articles by David Mertz assumes readers have a familiarity
with basic object-oriented programming concepts: inheritance, encapsulation, and
polymorphism. We pick up where the basics leave off; how some "exotic"
techniques can make applied programming tasks easier, more maintainable, and
just plain more elegant. This series will primarily present examples in Python,
but the concepts apply to other programming languages too.
This first installment examines metaclasses. Just as ordinary
instances (objects) are built out of classes, classes themselves are built out
of metaclasses. Most of the time, you do not need to worry about the implicit
metaclass (named type in Python) that is used in class
construction, but occasionally, deliberate control of a class' metaclass can
produce powerful effects. For example, metaclasses allow "aspect oriented
programming," meaning you can enhance classes with features like tracing
capabilities, object persistence, exception logging, and more.
An Object-Oriented Programming Review
In general principles, OOP works the same way across many programming
languages, modulo minor syntax differences. In an object-oriented programming
language, you can define classes that bundle together related data and
behavior. These classes can inherit some or all of their qualities from their
parents, but can also define attributes (data) or methods (behavior) of
their own. Classes generally act as templates for the creation of
instances (or simply objects). Different instances of the same
class typically have different data but share the same "shape"--e.g., the
Employee objects bob and jane both have a
.salary and a .room_number, but not the same room and
salary as each other.
A Metaprogramming Rejoinder
In Python (and other languages), classes are themselves objects that can be
passed around and introspected. Just as regular classes act as templates for
producing instances, metaclasses act as templates for producing classes.
Python has always had metaclasses. The metaclass machinery became exposed
much better with Python 2.2. Specifically, with version 2.2, Python stopped
being a language with just one special (mostly hidden) metaclass that created
every class object. Now, programmers can subclass the built-in metaclass
type and even dynamically generate classes with varying
metaclasses.
You do not need to use custom metaclasses to manipulate the production of
classes, however. A slightly less brain-melting concept is a class
factory. An ordinary function can return a class that is dynamically
created within the function body. In traditional Python syntax, you can write
the following.
Example 1. An old-fashioned Python 1.5.2 class factoryPython 1.5.2 (#0, Jun 27 1999, 11:23:01) [...]
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> def class_with_method(func):
...     class klass: pass
...     setattr(klass, func.__name__, func)
...     return klass
...
>>> def say_foo(self): print 'foo'
...
>>> Foo = class_with_method(say_foo)
>>> foo = Foo()
>>> foo.say_foo()
foo
The factory function class_with_method() dynamically creates and
returns a class that contains the method/function passed into the factory. The
class itself is manipulated within the function body before being returned. The
new module provides a more concise spelling, but
without the same options for custom code within the body of the class factory:
Example 2. A class factory in the new
module>>> from new import classobj
>>> Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>> Foo2().bar()
'bar'
>>> Foo2().say_foo()
foo
In all of these cases, the behaviors of the class (Foo,
Foo2) are not directly written as code, but are instead created by
calling functions at runtime, with dynamic arguments. Not only are the
instances dynamically created, so are the classes themselves.
From Class Factories to Metaclasses
Methods (i.e., of classes), like plain functions, can return objects. In that
sense, it is obvious that class factories can be classes just as easily as they
can be functions. In particular, Python 2.2+ provides a special class called
type that is just such a class factory. The new class
type is backwards-compatible with the older function of the same
name, by the way. The class type works as a class factory in the
same way that the function new.classobj does:
Example 3. type as a class factory metaclass>>> X = type('X',(),{'foo':lambda self:'foo'})
>>> X, X().foo()
(, 'foo')
Since type is now a (meta)class, you are free to subclass it:
Example 4. A type descendent as class factory>>> class ChattyType(type):
...     def __new__(cls, name, bases, dct):
...         print "Allocating memory for class", name
...         return type.__new__(cls, name, bases, dct)
...     def __init__(cls, name, bases, dct):
...         print "Init'ing (configuring) class", name
...         super(ChattyType, cls).__init__(name, bases, dct)
...
>>> X = ChattyType('X',(),{'foo':lambda self:'foo'})
Allocating memory for class X
Init'ing (configuring) class X
>>> X, X().foo()
(, 'foo')
The magic methods .__new__() and .__init__() are
special, but in conceptually the same way that they are for any other class. The
.__init__() method lets you configure the created object and the
.__new__() method lets you customize its allocation. The latter is
not widely overridden, but does exist for every Python 2.2 new-style class.
There is one feature of type descendents to be careful about; it
catches everyone who first plays with metaclasses. The first argument to methods
is conventionally called cls rather than self, because
the methods operate on the produced class, not the metaclass. Actually,
there is nothing special about this. All methods attach to their instances, and
the instance of a metaclass is a class. A better name makes this more obvious:
Example 5. Attaching class methods to produced classes>>> class Printable(type):
...     def whoami(cls): print "I am a", cls.__name__
...
>>> Foo = Printable('Foo',(),{})
>>> Foo.whoami()
I am a Foo
>>> Printable.whoami()
Traceback (most recent call last):
TypeError:  unbound method whoami() [...]
All of this surprisingly non-remarkable machinery comes with some syntax
sugar that both makes working with metaclasses easier and confuses new users.
There are several elements to the extra syntax. The resolution order of these
new variations is tricky though. Classes can inherit metaclasses from their
ancestors--notice that this is not the same thing as
having metaclasses as ancestors (another common confusion). For
old-style classes, defining a global __metaclass__ variable can
force a custom metaclass to be used. Most of the time, the safest approach is to
set a __metaclass__ class attribute for a class that wants to be
created via a custom metaclass. You must set the variable in the class
definition itself, since the metaclass is not used if the attribute is set later
(after the class object has already been created). For example:
Example 6. Setting a metaclass with a class attribute>>> class Bar:
...     __metaclass__ = Printable
...     def foomethod(self): print 'foo'
...
>>> Bar.whoami()
I am a Bar
>>> Bar().foomethod()
foo
Solving Problems with Magic
So far, we have seen the basics of metaclasses. Putting them to work is more
subtle. The challenge of using metaclasses is that in typical OOP design,
classes do not really do much. Class inheritance structures encapsulate
and package data and methods, but one typically works with instances in the
concrete.
There are two general categories of programming tasks where I think
metaclasses are genuinely valuable.
The first, and probably more common, category is where you do not know at
design time exactly what a class needs to do. Obviously, you will have
some idea about it, but some particular detail might depend on information that
will not be available until later. "Later" itself can be of two sorts: a), when
a library module is used by an application, and b), at runtime when some
situation exists. This category is close to what is often called "Aspect
Oriented Programming" (AOP). Let me show an elegant example:
Example 7. Metaclass configuration at runtime% cat dump.py
#!/usr/bin/python
import sys
if len(sys.argv) > 2:
    module, metaklass  = sys.argv[1:3]
    m = __import__(module, globals(), locals(), [metaklass])
    __metaclass__ = getattr(m, metaklass)
class Data:
    def __init__(self):
        self.num = 38
        self.lst = ['a','b','c']
        self.str = 'spam'
    dumps   = lambda self: `self`
    __str__ = lambda self: self.dumps()
data = Data()
print data
% dump.py

As you would expect, this application prints out a rather generic description
of the data object (a conventional instance). We get a rather
different result by passing runtime arguments to the application:
Example 8. Adding an external serialization metaclass% dump.py gnosis.magic MetaXMLPickler
  
  
  
The particular example uses the serialization style of
gnosis.xml.pickle, but the most current gnosis.magic
package also contains the metaclass serializers MetaYamlDump,
MetaPyPickler, and MetaPrettyPrint. Moreover, a user
of the dump.py "application" can impose the use of any
"MetaPickler" she wishes, from any Python package that defines one. Writing an
appropriate metaclass for this purpose will look something like this:
Example 9. Adding an attribute with a metaclassclass MetaPickler(type):
    "Metaclass for gnosis.xml.pickle serialization"
    def __init__(cls, name, bases, dict):
        from gnosis.xml.pickle import dumps
        super(MetaPickler, cls).__init__(name, bases, dict)
        setattr(cls, 'dumps', dumps)
The remarkable achievement of this arrangement is that the application
programmer need have no knowledge about what serialization will be used--nor
even whether serialization or some other cross-sectional capability will be
added at the command line.
Perhaps the most common use of metaclasses is similar to that of
MetaPicklers: adding, deleting, renaming, or substituting methods for those
defined in the produced class. In our example, a "native"
Data.dump() method is replaced by a different one from outside of
the application, at the time the class Data is created (and
therefore, in every subsequent instance).
Resources
A useful book on metaclasses is Putting Metaclasses to Work, by Ira
R. Forman, Scott Danforth, Addison-Wesley 1999 (ISBN 0201433052).
For metaclasses in Python specifically, Guido van Rossum's essay, " Unifying types and classes in
Python 2.2," is useful.
My Gnosis
Utilities package contains functions to make working with metaclasses easier
and more powerful.

[url=/pub/au/1190]David Mertz[/url]
, being a sort of Foucauldian
Berkeley, believes, esse est denunte.