Proposal: Wrapping Callable Objects Using Methods

Note: The proposal was accepted for inclusion in 1.5.2. See below for details.

Version: 0.4

Introduction

To fully understand the aim of this proposal, let's first explain the existing mechanism used by Python to implement the notion of instance methods.
When defining classes in Python we usually also add some function definitions to their definition body. These functions are special in that they cannot (normally) be accessed directly from the global namespace, but only through the classes namespace - they live in their classes namespace. Yet, they are quite normal Python functions, just like the ones defined in other scopes.
class A:
    def test(self,x):
        print id(self),x
a = A()
def func(x):
    print x
The difference between functions defined in class bodies (test()) and ones in the global namespace (func()) is that they are automatically wrapped by a special object whenever you access them through the class object (A.test) or a class instance (a.test()).
This special object is called a Python method. The main reason to have these wrappers is to provide an auxiliary storage for the extra object reference that is needed in order to make functions behave differently when being applied to the context of a class instances. Basically, Python methods hold three references:

the object reference (which may be None)

the function implementing the functionality

the class reference

A method can be bound (the object reference points to a class instance) or unbound (the object reference is None).
Magic binding
Functions accessed through the class are turned into unbound methods, ones accessed through the class instances (a.test) are wrapped into bound methods. Now, when calling a bound method with a set of arguments (a.test(42)), the object reference is prepended to the argument set and the function then called with the resulting tuple (A.test(a,42)).
As an addition to this scheme, unbound methods accessed through an instance are also turned into bound methods (e.g. a.test). The latter feature is what we are going to focus on below.
This magic is very nice and greatly reduces programming effort, but unfortunately does not extend well to other applications, since it only happens to Python functions and not all callable objects, e.g. you cannot use functions that are written in C as building parts for Python methods.

General Idea

Basically only these two things are necessary to be able to re-implement Python methods in C for speed:

Extend Python methods to not only store references to Python functions, but to any callable Python object.

Provide a way to construct unbound methods using arbitrary callable methods as underlying executable object.

No extensions to the magic wrapping process are done, since this would break code. Instead, we leave everything as it is, because unbound methods are already subject to the above late binding mechanism (they are turned into bound methods when accessed through an instance object) and thus make the callable object receive the same arguments as a Python function would in that situation.

Technical Implementation

The implementation will have to resolve three issues:

nearly 100% backward compatibility

small footprint

simple usage

All three are manageable:
Compatibility
Extending Python methods to wrap any callable Python object means that code using the im_func attribute of methods directly may no longer expect to find a genuine Python function (having func_* attributes). The Python library is not harmed in any way by this, and it is not very likely that any application, apart from some debuggers maybe, rely on this "feature".
Since no extended magic is performed, only those objects will turn into unbound methods that you explicitly wrap using an unbound method constructor. Other code that places callable objects into class namespaces will continue to work as before, because the wrapping process still only effects Python methods and Python functions.
What about code complexity ?
Making Python methods wrap callables is not too difficult, since only the representation method (used by repr()) needs to be fixed. Adding the new constructor to the new module is also only a minor patch.
Only some of the code in ceval.c has to be tweaked a little. This is because the existing call_function() code assumes to find a Python function instead of an arbitrary callable object. But it's a nice chance to clean up the calling code a little and demangle the schemes used.
How do I create unbound methods ?
There already is a special module that provides constructors for many different special Python objects. It is called new and has to be explicitly enabled in Modules/Setup by hand to be compiled and installed. This module has to be modified to allow any callable object to be passed as implementing function. Additionally, a new constructor for unbound methods must be added.
Using the module, you can then add your own unbound methods to any Python class:
from new import unboundmethod
# Define class A
class A:
    a = 2
    b = 3
    def pyfunc(self,*args):
        return '%s called with %s' % (self,args)
# Since the class A is needed for wrapping the C function, we have
# to install the function as method *after* the class is defined.
# We use the builtin function getattr() here as demo function,
# since it is always available, any builtin function will do
# though: the instance object is passed as first argument.
A.get = unboundmethod(getattr,A)
# More examples:
A.set = unboundmethod(setattr,A)
A.hash = unboundmethod(hash,A)
# Instantiate the class
o = A()
print o.a
print o.pyfunc(1,2,3)
print o.get('a')                # this will call getattr(o,'a')
o.set('a',3)                    # this will call setattr(o,'a',3)
print o.get('a')                # this will call getattr(o,'a')
print o.hash()                  # this will call hash(o)
You can also use a more direct way:
class Base:
    pass
class Class(Base):
    a = 2
    b = 3
    get = unboundmethod(getattr,Base)
    set = unboundmethod(setattr,Base)
    hash = unboundmethod(hash,Base)
# Instantiate the class
o = Class()
print o.a
print o.get('a')                # this will call getattr(o,'a')
o.set('a',3)                    # this will call setattr(o,'a',3)
print o.get('a')                # this will call getattr(o,'a')
print o.hash()                  # this will call hash(o)
This works because you only have to assure that the unbound method's
class is a base class of the instances class.
If you plan to only include some builtin methods in you Python class or want to enable the faster ones by adding a mixin class that overrides the Python methods with your builtin methods, you can use the template functions from the included module BuiltinClass.py:
from BuiltinClass import BuiltinMixinClass
MixinFastMethods = \
         BuiltinMixinClass(getattr,setattr,delattr,hasattr)
class Class(MixinFastMethods):
    a = 2
    b = 3
    def __init__(self):
        self.data = {'a':4}
    # override a C method with a Python method
    def getattr(self,x):
        return self.data[x]
    # add more methods written in Python
    def keys(self):
        return self.data.keys()
# Instantiate the class
o = Class()
print o.a
print o.getattr('a') # this will call the Python method
o.setattr('a',3) # this will call setattr(o,'a',3)
print o.hasattr('a') # this will call hasattr(o,'a')
For the meta-class fans, here's another method that uses Donald Beaudry's object extension module (there's also a README providing additional information):
import object
from new import unboundmethod
class A(object.base):
    class __class__:
        def __init__(self, name, bases, dict):
             self.get = unboundmethod(getattr, self)
             self.set = unboundmethod(setattr, self)
             self.hash = unboundmethod(hash, self)
    a = 2
    b = 3
    def pyfunc(self, *args):
        return '%s called with %s' % (self, args)
The same is probably also doable with the meta-class protocol that Guido explains in his meta-class essay. This is left as exercise to the reader ;-)
Unfortunately, some of the special methods are not assignable after creation. These are

__getattr__(self,what)

__setattr__(self,what,to)

__delattr__(self,what)

Especially the __getattr__ would be a good candidate for a rewrite in C in case performance suffers too much from using this very convenient hook. Thus, it makes sense to allow setting these special attributes in unrestricted mode too.
OK, I like the proposal, now who's going to write the code ?
The proposal has already been implemented. All you have to do is download the patch and apply it (according to the included instructions) to a clean Python 1.5.1 source distribution. Please post any feedback either to me or the newsgroup.

Note: Guido has accepted the proposal with some slight changes to the way you create unbound methods. See Misc/NEWS in the 1.5.2 source distribution to find out more about it. The extra changes to ceval.c mentioned below were not included.

Anything else in the box ?
The 0.3 version of the patch is a stripped down to the bone implementation of the above mentioned features. The 0.2 version is identical in terms of functionality, but also includes some other goodies:
Changes in ceval.c:
I am curious to here your opinions on this one. IMHO, the scheme becomes much more transparent when implemented in this way. Basically every case is turned into a separate function:

/* Calling scheme switch */
if (PyCFunction_Check(func))
      result = call_c_function(func, arg, kw);
else if (PyFunction_Check(func))
      result = call_function(func, arg, kw);
else if (PyClass_Check(func))
      result = PyInstance_New(func, arg, kw);
else if (PyInstance_Check(func))
      result = call_instance(func, arg, kw);
else if (PyMethod_Check(func))
      result = call_method(func, arg, kw);
else if ((call = func->ob_type->tp_call) != NULL)
      result = (*call)(func, arg, kw);
else {
      PyErr_SetString(PyExc_TypeError,
                      "call of non-function");
      return NULL;
}
The patch also enables some macros for faster access to internal method attributes - I just couldn't see any reason for three function calls that implement three unnecessary type checks. If you would like to do some benchmarks with/without the patch applied, try pybench or pystone.
As third aside, the special class attributes:

__getattr__

__setattr__

__delattr__

__name__

__dict__

__bases__

are made assignable in unrestricted mode. You can assign any callable object to the three attribute methods, strings to __name__, dictionaries to __dict__ and tuples of class objects to __bases__. Special care is taken that assignment to __bases__ does not introduce circular inheritance references. These would cause segmentation faults due to an exhausted stack. Note: These will appear in 1.5.2.
Things that still need to be done:

clear up this document a little

convince Guido of the usefulness of these changes and have them included in Python 1.5.2 [Done]

rewrite the document as documentation for the new feature

Contact : Impressum : Terms & Conditions : Privacy Policy : Trademarks 2025-02-22