16.7 Implementing C Function Callbacks to a Python Function

Credit: Swaminathan Narayanan

16.7.1 Problem

You must call a C function that takes a function callback as an argument, and you want to pass a Python function as the callback.

16.7.2 Solution

For this, we must wrap the Python function in a C function to be passed as the actual C-level callback. For example:

#include "python.h"

/* the C standard library qsort function, just as an example! */
extern void qsort(void *, size_t, size_t, int (*)(const void *, const void *));

/* static data (sigh), as we have no callback data in this (nasty) case */
static PyObject *py_compare_func = NULL;

static int
stub_compare_func(const void *cva, const void *cvb)
{
    int retvalue = 0;
    const PyObject **a = (const PyObject**)cva;
    const PyObject **b = (const PyObject**)cvb;

    // Build up the argument list...
    PyObject *arglist = Py_BuildValue("(OO)", *a, *b);

    // ...for calling the Python compare function
    PyObject *result = PyEval_CallObject(py_compare_func, arglist);

    if (result && PyInt_Check(result)) {
        retvalue = PyInt_AsLong(result);
    }

    Py_XDECREF(result);
    Py_DECREF(arglist);

    return retvalue;
}

static PyObject *pyqsort(PyObject *obj, PyObject *args)
{
    PyObject *pycompobj;
    PyObject *list;
    if (!PyArg_ParseTuple(args, "OO", &list, &pycompobj))
        return NULL;

    // Make sure second argument is a function
    if (!PyCallable_Check(pycompobj)) {
        PyErr_SetString(PyExc_TypeError, "Need a callable object!");
    } else {
        // Save the compare function. This obviously won't work for multithreaded
        // programs and is not even a reentrant, alas -- qsort's fault!
        py_compare_func = pycompobj;
        if (PyList_Check(list)) {
            int size = PyList_Size(list);
            int i;

            // Make an array of (PyObject *), because qsort does not know about
            // the PyList object
            PyObject **v = (PyObject **) malloc( sizeof(PyObject *) * size );
            for (i=0; i<size; ++i) {
                v[i] = PyList_GetItem(list, i);
                // Increment the reference count, because setting the list 
                // items below will decrement the reference count
                Py_INCREF(v[i]);
            }
            qsort(v, size, sizeof(PyObject*), stub_compare_func);
            for (i=0; i<size; ++i) {
                PyList_SetItem(list, i, v[i]);
                // need not do Py_DECREF - see above
            }
            free(v);
        }
    }
    Py_INCREF(Py_None);
    return Py_None;
}

static PyMethodDef qsortMethods[] = {
    { "qsort", pyqsort, METH_VARARGS },
    { NULL, NULL }
};

_ _declspec(dllexport) void initqsort(void) {
    PyObject *m;
    m = Py_InitModule("qsort", qsortMethods);
}

16.7.3 Discussion

Let's say you have a function in C or C++ that takes a function callback as an argument. You want to call this function and pass a Python function as the callback. For example, you want to call the standard C library function qsort on a suitably arrayized Python list and pass a Python function as the comparison function:

>>> import qsort
>>> a = [9, 3, 5, 4, 1]
>>> def revcmp(a, b): return cmp(b, a)
...
>>> qsort.qsort(a, revcmp)
>>> a
[9, 5, 4, 3, 1]

Of course, this is strictly for demonstration purposes, since Python's own sort list method is far better!

When extending Python, you may come across existing C functions that take a function callback. It makes sense to pass Python a function as the callback function. The trick is to have a C function callback call the Python function by suitably marshaling the arguments. This is done by stub_compare_func in the recipe. Py_BuildValue is used to pass the two Python objects being compared back to the Python function.

In the case of qsort, there is no user data that can be passed, which is usually the callback convention. This means that we have to store the Python function in a static variable and use it to call in the C callback. This is not an ideal situation, given that it would not work in a multithreaded, or otherwise reentrant, program). While there is no solution for this particular case (as far as I know), the usual trick is to pass the Python function as user data to the function callback, an approach that is reentrant and thread-safe. But the possibility of using this better approach depends on whether the C-level callback architecture is well-designed.

This recipe's wrapper of qsort copies the PyObject pointers to a separate array that is sorted using the C library's qsort. The pointers are then put back in the original list. The reference counts of the items in the list that are being replaced are decreased behind the scenes. However, this is okay, because we increased them beforehand. Consequently, we do not need to do a Py_DECREF after setting the item in the list. Thus, this recipe also serves nicely as an example of a reference count handling quirk.

16.7.4 See Also

The Extending and Embedding manual is available as part of the standard Python documentation set at http://www.python.org/doc/current/ext/ext.html; documentation on the Python C API at http://www.python.org/doc/current/api/api.html; documentation on qsort for your standard C library.