topic

Programming of metaclasses on Python

Raising of object-oriented programming on a new level

The majority of readers already familiarly with concepts of object-oriented programming: inheritance, inkapsuljaciej, polymorphism. But creation of objects of the set class with the certain parents is usually represented by initially set operation. It appears, that a number{line} of new program designs becomes or more simple, or in general though a little possible{probable} if you can adjust process of creation of objects. Metaclasses resolve the certain types of " aspektno-guided programming "; for example, it is possible to strengthen classes such characteristics, as opportunities of trace, objective persistentnost`, registration of exceptions and so on.

The brief review of object-oriented programming (OOP)

Give, having spent half-minutes, we shall recollect, that such OOP. In language of object-oriented programming it is possible to define{determine} classes, which problem{task} - to unit the connected data and behaviour. These classes can inherit some or all properties of the parents, they also define{determine} own attributes (data) or methods (behaviour). In result, classes, as a rule, represent itself as patterns for creation of copies (which from time to time name simply objects). Various copies of the same class usually have the different data though they will be submitted in an identical kind, for example, at both objects of a class 'Employee' bob and jane is .salary and .room_number, but values room (room) and salary (salary) at everyone are various.

Some object-oriented programming languages, switching Python, provide introspective (or reflective) objects. In other words, the introspective object can describe itself: to what class this copy belongs? Who ancestors of this class? What methods and attributes are accessible to object? Introspekcija allows function or a method managing objects, to make of the decision, being based on what kind of object is passed. Even without introspekcii functions often vetvjatsja, leaning{basing} on the data of a copy - for example, the route to jane.room_number differs from a way to bob.room_number, as they in " various rooms " (values room at them are various). With the help introspekcii also it is possible to calculate correctly bonus (bonus) jane, having passed{missed} this calculation for bob, for example, because at jane there is an attribute .profit_share or that bob is a copy of derivative class Hourly (Employee).

The "Metaprogram" answer

The base system OOP outlined above, is enough powerful. However, in this description one moment has not received due attention: in Python (and other languages) classes are objects which it is possible to pass and subject introspekcii. But as objects how was marked, created with use of classes as patterns that is a pattern for creation of classes? Certainly, metaclasses.

In Python always there were metaclasses. However, the technology involved in metaclasses, became much more obvious with output{exit} Python 2.2. Namely, in version 2.2 Python has ceased to be language only with one special (usually invisible) metaclass which created each object of a class. Now programmers can be inherited from the built - in metaclass type and even dynamically to generate classes with various metaclasses. Certainly, only that you can manipulate metaclasses on Python 2.2, yet does not explain, what for to you it.

Moreover, you do not need to use the metaclasses determined by the user to operate creation of classes. A little less golovolomnaja the concept - factory of classes (class factory): ordinary function can return a class which has been dynamically created within the limits of a body of function. In traditional syntax Python you can write:

Listing 1. Traditional factory of classes on Python 1.5.2

Python 1.5.2 (*0, Jun 271999, 11:23:01) [...]

>>> def class_with_method (func):

... class klass: pass

... setattr (klass, func. __ name __, func)

... return klass

...

>>> def say_foo (self): print 'foo'

...

>>> Foo = class_with_method (say_foo)

>>> foo = Foo ()

>>> foo.say_foo ()

foo

Function of factory class_with_method () dynamically creates and returns a class containing method / function, transmitted in this factory. The class is processed within the limits of a body of function before he is returned. The module new provides more laconic expression, but without an opportunity of definition by the user of an additional code within the limits of a body of factory of classes. For example:

Listing 2. Factory of classes in the module new

>>> from new import classobj

>>> Foo2 = classobj ('Foo2', (Foo,), {' bar ':lambda self:'bar '})

>>> Foo2 () .bar ()

'bar'

>>> Foo2 () .say_foo ()

foo

In all these cases the behaviour of a class (Foo, Foo2) is not written down directly as a code, and created by means of a call during execution{performance} of functions with calculated arguments. It is necessary to emphasize, that classes are dynamically created, instead of is simple copies.

Metaclasses: the decision demanding a problem?

" Metaclasses - the big magic, than 99 % of users are necessary. If you asked a question, whether they are necessary for you, means, they are not necessary for you (those for whom they are really necessary, precisely it know, and they do not need an explanation, what for) ".

Tim Piters (Tim Peters), the largest authority on area Python

Methods (classes), as well as usual functions, can return objects. In this sense obviously, that factories of classes so can be classes, as well as functions. In particular, Python 2.2 + gives the special class named type which such factory of classes and is. Certainly, readers learn{find out} in type () less pretentious built - in function of earlier versions Python - fortunately, the behaviour of old function type () is supported by a class type (in other words, type (obj) returns a type / class of object obj). The new class works as factory of classes the same way as function new.classobj before did{made}:

Listing 3. type as a metaclass of factory of classes

>>> X = type ('X', (), {' foo ':lambda self:'foo '})

>>> X, X () .foo ()

(<class ' __ main __. X '> ', foo ')

But as now type is (meta) a class, you can create from him{it} a derivative class:

Listing 4. The descendant type as factory of classes

>>> class ChattyType (type):

... def __ new __ (cls, name, bases, dct):

... print " Allocating memory for class ", name

... return type. __ new __ (cls, name, bases, dct)

... def __ init __ (cls, name, bases, dct):

... print " Init'ing (configuring) class ", name

... super (ChattyType, cls). __ init __ (name, bases, dct)

...

>>> X = ChattyType ('X', (), {' foo ':lambda self:'foo '})

Allocating memory for class X

Init'ing (configuring) class X

>>> X, X () .foo ()

(<class ' __ main __. X '>', foo)

Magic methods. __ new __ () and. __ init __ () they are special, but conceptually same, as well as at any other class. A method. __ init __ () allows to configure the created object; a method. __ new __ () allows to adjust his{its} creation. Last, certainly, is not used widely, but exists for each class of new style Python 2.2 (it is usually inherited, instead of is substituted).

Descendants type have one property which is necessary for taking into account; on him everyone who for the first time uses metaclasses are caught. The first argument in methods usually is called cls, instead of self as these methods process the created class, instead of a metaclass. Actually, in it there is nothing especial; all methods communicate with svoimiehkzempljarami, and the copy of a metaclass is a class. The nonspecial name does{makes} it more obvious [1]:

Listing 5. An attachment of methods of a class to the created classes

>>> class Printable (type):

... def whoami (cls): print " I am a ", cls. __ name __

...

>>> Foo = Printable ('Foo', (), {}}

>>> Foo.whoami ()

I am a Foo

>>> Printable.whoami ()

Traceback (most recent call last):

TypeError: unbound method whoami () [...]

All this surprisingly inconspicuous technology is accompanied by the certain syntactic ornaments simplifying job with metaclasses and simultaneously confusing new users. In this additional syntax there are some elements. The order of interpretation of these new variations odd. Classes can inherit metaclasses from the ancestors - notice, what is it not same, that presence of metaclasses as ancestors (one more usual error). For classes of old style definition of a global variable __ metaclass __ results in use metaklasssa, certain{determined} by the user. However generally the most safe approach - to set attribute of a class __ metaclass __ for a class which wants to be created on a metaclass determined by the user. You should set this variable in the definition of a class as the metaclass is not used if this attribute is set later (after the object of a class has already been created). For example:

Listing 6. The task of a metaclass with attribute of a class

>>> class Bar:

... __ metaclass __ = Printable

... def foomethod (self): print 'foo'

...

>>> Bar.whoami ()

I am a Bar

>>> Bar () .foomethod ()

foo

The problems decided{solved} by magic

Till now we considered{examined} bases of metaclasses. However real application of metaclasses has the subtleties. Complexity with use of metaclasses consists that in typical model OOP classes actually do{make} not much. The structure of inheritance of classes is convenient for inkapsuljacii and associations of the data and methods, but copies are usually really used.

There are two general{common} categories of problems{tasks} of programming for which, in our opinion, metaclasses are extremely useful.

The first, and it is probable more the general{common} case, when during designing you do not know it precisely, that it is necessary to do{make} to a class. It is obvious, that at you performance about it will appear, but the certain certain detail can depend on the information which becomes accessible later. Itself can be two kinds "later": (a) when the library module will be used by the application; (b) during execution{performance} when there will be a certain situation. This the category is close to that often name " Aspektno-guided programming " (AOP). We shall show, that we mean on the following elegant example:

Listing 7. Konfigurirovanie a metaclass during execution{performance}

% cat dump.py

*!/usr/bin/python

import sys

if len (sys.argv)> 2:

module, metaklass = sys.argv [1:3]

m = __ import __ (module, globals (), locals (), [metaklass])

__ metaclass __ = getattr (m, metaklass)

class Data:

def __ init __ (self):

self.num = 38

self.lst = ['a', 'b', 'c']

self.str = 'spam'

dumps = lambda self: 'self'

__ str __ = lambda self: self.dumps ()

data = Data ()

print data

% dump.py

<__ main __. Data instance at 1686a0>

As you could expect, this application deduces rather the general{common} description of object data (conditional object of a copy). However, if arguments of time of execution{performance} are passed in the application, it is possible to receive a little bit excellent{different} result:

Listing 8. Addition of a metaclass of external serialization

% dump.py gnosis.magic MetaXMLPickler

<? xml version = " 1.0"?>

<! DOCTYPE PyObject SYSTEM "PyObjects.dtd">

</attr>

</PyObject>

In this private{individual} example style of serialization gnosis.xml.pickle is applied, but the current version of a package gnosis.magic also contains metaclasses serializatorov MetaYamlDump, MetaPyPickler and MetaPrettyPrint. Besides the user of "application" dump.py can demand use of any desirable "MetaPickler" from any package Python which defines{determines} it{him}. The corresponding metaclass intended for this purpose, will approximately look so:

Listing 9. Addition of attribute with a metaclass

class MetaPickler (type):

" Metaclass for gnosis.xml.pickle serialization "

def __ init __ (cls, name, bases, dict):

from gnosis.xml.pickle import dumps

super (MetaPickler, cls). __ init __ (name, bases, dict)

setattr (cls, ' dumps', dumps)

Remarkable achievement of this approach consists that the applied programmer does not need to know anything about, whether what serialization will be used even serialization or other way of external performance will be added in the command line.

Probably, most general{common} use of metaclasses is similar to application MetaPicklers: addition, removal{distance}, renaming or substitution of methods instead of the methods determined in the created class. In our example the "built - in" method Data.dump () is replaced with other method, external in relation to the application, during creation of class Data (and, hence, in each subsequent copy).

Other ways of the decision of problems with the help of magic

There is an area of programming where classes are frequently more important, than copies. For example, declarative mini-languages (declarative mini-languages) are libraries Python which program logic is expressed directly in the announcement of a class. David considers{examines} them in clause{article} " Creation of declarative mini-languages " (Create declarative mini-languages). In similar cases use of metaclasses for influence on process of creation of a class can be rather effective.

One of the declarative libraries based on classes, is gnosis.xml.validity. In this structure you declare a number{line} of " classes of an admissibility " (" validity classes ") which describe a set of restrictions for allowable documents XML. These announcements are very close to those, that contain in descriptions such as the document (DTDs). For example, the document "dissertation" can be configured with the help of the following code:

Listing 10. Rules gnosis.xml.validity in simple_diss.py

from gnosis.xml.validity import *

class figure (EMPTY): pass

class _mixedpara (Or): _disjoins = (PCDATA, figure)

class paragraph (Some): _type = _mixedpara

class title (PCDATA): pass

class _paras (Some): _type = paragraph

class chapter (Seq): _order = (title, _paras)

class dissertation (Some): _type = chapter

If to try to create a copy dissertation without appropriate subelements, the exception describing a situation is raised; similar takes place for each subelement. Correct subelements will be automatically generated from more simple arguments if there is only one consistent way "to complete" type up to a correct status.

Though classes of an admissibility often are (informally based on preliminary existing DTD, copies of these classes are printed as vnekontekstnye (unadorned) fragments of document XML, for example:

Listing 11. Creation of the document with a base class of an admissibility

>>> from simple_diss import *

>>> ch = LiftSeq (chapter, (' It Starts','When it began '))

>>> print ch

<chapter> <title> It Starts </title>

<paragraph> When it began </paragraph>

</chapter>

Using a metaclass for creation of classes of an admissibility, we can generate DTD from announcements of a class (and thus to add an additional method in these classes):

Listing 12. Use of metaclasses during import of the module

>>> from gnosis.magic import DTDGenerator,

... import_with_metaclass,

... from_import

>>> d = import_with_metaclass (' simple_diss', DTDGenerator)

>>> from_import (d, '*')

>>> ch = LiftSeq (chapter, (' It Starts','When it began '))

>>> print ch.with_internal_subset ()

<? xml version = '1.0'?>

<! DOCTYPE chapter [

<! ELEMENT figure EMPTY>

<! ELEMENT dissertation (chapter) +>

<! ELEMENT chapter (title, paragraph +)>

<! ELEMENT title (*PCDATA)>

<! ELEMENT paragraph ((*PCDATA|figure)) +>

<chapter> <title> It Starts </title>

<paragraph> When it began </paragraph>

</chapter>

The package gnosis.xml.validity anything does not know about DTD and internal subsets. These concepts and opportunities are entirely submitted by metaclass DTDGenerator without entering any changes in gnosis.xml.validity or simple_diss.py. DTDGenerator does not substitute an own method. __ str __ () in classes which he creates - you still can deduce{remove} vnekontekstnyj fragment XML - but the metaclass could modify similar magic methods easily.

Metaadvantages

The package gnosis.magic contains some utilities for job with metaclasses, and also some examples of metaclasses which can be applied in aspektno-guided programming. Most important of these utilities - import_with_metaclass (). This function involved in the previous example, allows to import the any module, creating all classes of this module with use of the metaclass determined by the user, instead of type. What a new opportunity you have wanted to set in this module, she can be determined in a metaclass which you create (or receive). gnosis.magic contains some connected metaclasses of serialization; other package could include opportunities of trace, objective persistentnost`, registration of exceptions or something else.

Function import_with_metaclass () illustrates some opportunities of programming of metaclasses:

Listing 13. Function import_with_metaclass () from [gnosis.magic]

def import_with_metaclass (modname, metaklass):

" Module importer substituting custom metaclass "

class Meta (object): __ metaclass __ = metaklass

dct = {' __ module __ ':modname}

mod = __ import __ (modname)

for key, val in mod. __ dict __.items ():

if inspect.isclass (val):

setattr (mod, key, type (key, (val, Meta), dct))

return mod

In this function it is necessary to pay attention to style - ordinary class Meta is created with use of the set metaclass. But as soon as Meta it is added as an ancestor, his{its} descendants also are created with the help of this metaclass. Basically, the class similar Meta, could give both the generator of a metaclass, and a number{line} of inherited methods - these two aspects of inheritance are orthogonal.