Author: Valentin Haenel
This chapter contains an introduction to the many different routes for making your native code (primarily C/C++) available from Python, a process commonly referred to wrapping. The goal of this chapter is to give you a flavour of what technologies exist and what their respective merits and shortcomings are, so that you can select the appropriate one for your specific needs. In any case, once you do start wrapping, you almost certainly will want to consult the respective documentation for your selected technique.
Chapters contents
This chapter covers the following techniques:
These four techniques are perhaps the most well known ones, of which Cython is probably the most advanced one and the one you should consider using first. The others are also important, if you want to understand the wrapping problem from different angles. Having said that, there are other alternatives out there, but having understood the basics of the ones above, you will be in a position to evaluate the technique of your choice to see if it fits your needs.
The following criteria may be useful when evaluating a technology:
Before you set out, you should consider your use case. When interfacing with native code, there are usually two use-cases that come up:
Each technology is demonstrated by wrapping the cos function from math.h. While this is a mostly a trivial example, it should serve us well to demonstrate the basics of the wrapping solution. Since each technique also includes some form of Numpy support, this is also demonstrated using an example where the cosine is computed on some kind of array.
Last but not least, two small warnings:
The Python-C-API is the backbone of the standard Python interpreter (a.k.a CPython). Using this API it is possible to write Python extension module in C and C++. Obviously, these extension modules can, by virtue of language compatibility, call any function written in C or C++.
When using the Python-C-API, one usually writes much boilerplate code, first to parse the arguments that were given to a function, and later to construct the return type.
Advantages
Disadvantages
Note
The Python-C-Api example here serves mainly for didactic reasons. Many of the other techniques actually depend on this, so it is good to have a high-level understanding of how it works. In 99% of the use-cases you will be better off, using an alternative technique.
Note
Since refernce counting bugs are easy to create and hard to track down, anyone really needing to use the Python C-API should read the section about objects, types and reference counts from the official python documentation. Additionally, there is a tool by the name of cpychecker which can help discover common errors with reference counting.
The following C-extension module, make the cos function from the standard math library available to Python:
/* Example of wrapping cos function from math.h with the Python-C-API. */
#include <Python.h>
#include <math.h>
/* wrapped cosine function */
static PyObject* cos_func(PyObject* self, PyObject* args)
{
double value;
double answer;
/* parse the input, from python float to c double */
if (!PyArg_ParseTuple(args, "d", &value))
return NULL;
/* if the above function returns -1, an appropriate Python exception will
* have been set, and the function simply returns NULL
*/
/* call cos from libm */
answer = cos(value);
/* construct the output from cos, from c double to python float */
return Py_BuildValue("f", answer);
}
/* define functions in module */
static PyMethodDef CosMethods[] =
{
{"cos_func", cos_func, METH_VARARGS, "evaluate the cosine"},
{NULL, NULL, 0, NULL}
};
/* module initialization */
PyMODINIT_FUNC
initcos_module(void)
{
(void) Py_InitModule("cos_module", CosMethods);
}
As you can see, there is much boilerplate, both to «massage» the arguments and return types into place and for the module initialisation. Although some of this is amortised, as the extension grows, the boilerplate required for each function(s) remains.
The standard python build system distutils supports compiling C-extensions from a setup.py, which is rather convenient:
from distutils.core import setup, Extension
# define the extension module
cos_module = Extension('cos_module', sources=['cos_module.c'])
# run the setup
setup(ext_modules=[cos_module])
This can be compiled:
$ cd advanced/interfacing_with_c/python_c_api
$ ls
cos_module.c setup.py
$ python setup.py build_ext --inplace
running build_ext
building 'cos_module' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/include/python2.7 -c cos_module.c -o build/temp.linux-x86_64-2.7/cos_module.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/cos_module.o -L/home/esc/anaconda/lib -lpython2.7 -o /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/python_c_api/cos_module.so
$ ls
build/ cos_module.c cos_module.so setup.py
The file cos_module.so contains the compiled extension, which we can now load in the IPython interpreter:
In [1]: import cos_module
In [2]: cos_module?
Type: module
String Form:<module 'cos_module' from 'cos_module.so'>
File: /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/python_c_api/cos_module.so
Docstring: <no docstring>
In [3]: dir(cos_module)
Out[3]: ['__doc__', '__file__', '__name__', '__package__', 'cos_func']
In [4]: cos_module.cos_func(1.0)
Out[4]: 0.5403023058681398
In [5]: cos_module.cos_func(0.0)
Out[5]: 1.0
In [6]: cos_module.cos_func(3.14159265359)
Out[7]: -1.0
Now let’s see how robust this is:
In [10]: cos_module.cos_func('foo')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-10-11bee483665d> in <module>()
----> 1 cos_module.cos_func('foo')
TypeError: a float is required
Analog to the Python-C-API, Numpy, which is itself implemented as a C-extension, comes with the Numpy-C-API. This API can be used to create and manipulate Numpy arrays from C, when writing a custom C-extension. See also: :ref:`advanced_numpy`_.
Note
If you do ever need to use the Numpy C-API refer to the documentation about Arrays and Iterators.
The following example shows how to pass Numpy arrays as arguments to functions and how to iterate over Numpy arrays using the (old) Numpy-C-API. It simply takes an array as argument applies the cosine function from the math.h and returns a resulting new array.
/* Example of wrapping the cos function from math.h using the Numpy-C-API. */
#include <Python.h>
#include <numpy/arrayobject.h>
#include <math.h>
/* wrapped cosine function */
static PyObject* cos_func_np(PyObject* self, PyObject* args)
{
PyArrayObject *in_array;
PyObject *out_array;
NpyIter *in_iter;
NpyIter *out_iter;
NpyIter_IterNextFunc *in_iternext;
NpyIter_IterNextFunc *out_iternext;
/* parse single numpy array argument */
if (!PyArg_ParseTuple(args, "O!", &PyArray_Type, &in_array))
return NULL;
/* construct the output array, like the input array */
out_array = PyArray_NewLikeArray(in_array, NPY_ANYORDER, NULL, 0);
if (out_array == NULL)
return NULL;
/* create the iterators */
in_iter = NpyIter_New(in_array, NPY_ITER_READONLY, NPY_KEEPORDER,
NPY_NO_CASTING, NULL);
if (in_iter == NULL)
goto fail;
out_iter = NpyIter_New((PyArrayObject *)out_array, NPY_ITER_READWRITE,
NPY_KEEPORDER, NPY_NO_CASTING, NULL);
if (out_iter == NULL) {
NpyIter_Deallocate(in_iter);
goto fail;
}
in_iternext = NpyIter_GetIterNext(in_iter, NULL);
out_iternext = NpyIter_GetIterNext(out_iter, NULL);
if (in_iternext == NULL || out_iternext == NULL) {
NpyIter_Deallocate(in_iter);
NpyIter_Deallocate(out_iter);
goto fail;
}
double ** in_dataptr = (double **) NpyIter_GetDataPtrArray(in_iter);
double ** out_dataptr = (double **) NpyIter_GetDataPtrArray(out_iter);
/* iterate over the arrays */
do {
**out_dataptr = cos(**in_dataptr);
} while(in_iternext(in_iter) && out_iternext(out_iter));
/* clean up and return the result */
NpyIter_Deallocate(in_iter);
NpyIter_Deallocate(out_iter);
Py_INCREF(out_array);
return out_array;
/* in case bad things happen */
fail:
Py_XDECREF(out_array);
return NULL;
}
/* define functions in module */
static PyMethodDef CosMethods[] =
{
{"cos_func_np", cos_func_np, METH_VARARGS,
"evaluate the cosine on a numpy array"},
{NULL, NULL, 0, NULL}
};
/* module initialization */
PyMODINIT_FUNC
initcos_module_np(void)
{
(void) Py_InitModule("cos_module_np", CosMethods);
/* IMPORTANT: this must be called */
import_array();
}
To compile this we can use distutils again. However we need to be sure to include the Numpy headers by using :func:numpy.get_include.
from distutils.core import setup, Extension
import numpy
# define the extension module
cos_module_np = Extension('cos_module_np', sources=['cos_module_np.c'],
include_dirs=[numpy.get_include()])
# run the setup
setup(ext_modules=[cos_module_np])
To convince ourselves if this does actually works, we run the following test script:
import cos_module_np
import numpy as np
import pylab
x = np.arange(0, 2 * np.pi, 0.1)
y = cos_module_np.cos_func_np(x)
pylab.plot(x, y)
pylab.show()
And this should result in the following figure:
Ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.
Advantages
Disadvantages
As advertised, the wrapper code is in pure Python.
""" Example of wrapping cos function from math.h using ctypes. """
import ctypes
from ctypes.util import find_library
# find and load the library
libm = ctypes.cdll.LoadLibrary(find_library('m'))
# set the argument type
libm.cos.argtypes = [ctypes.c_double]
# set the return type
libm.cos.restype = ctypes.c_double
def cos_func(arg):
''' Wrapper for cos from math.h '''
return libm.cos(arg)
We may now use this, as before:
In [1]: import cos_module
In [2]: cos_module?
Type: module
String Form:<module 'cos_module' from 'cos_module.py'>
File: /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/ctypes/cos_module.py
Docstring: <no docstring>
In [3]: dir(cos_module)
Out[3]:
['__builtins__',
'__doc__',
'__file__',
'__name__',
'__package__',
'cos_func',
'ctypes',
'find_library',
'libm']
In [4]: cos_module.cos_func(1.0)
Out[4]: 0.5403023058681398
In [5]: cos_module.cos_func(0.0)
Out[5]: 1.0
In [6]: cos_module.cos_func(3.14159265359)
Out[6]: -1.0
As with the previous example, this code is somewhat robust, although the error message is not quite as helpful, since it does not tell us what the type should be.
In [7]: cos_module.cos_func('foo')
---------------------------------------------------------------------------
ArgumentError Traceback (most recent call last)
<ipython-input-7-11bee483665d> in <module>()
----> 1 cos_module.cos_func('foo')
/home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/ctypes/cos_module.py in cos_func(arg)
12 def cos_func(arg):
13 ''' Wrapper for cos from math.h '''
---> 14 return libm.cos(arg)
ArgumentError: argument 1: <type 'exceptions.TypeError'>: wrong type
Numpy contains some support for interfacing with ctypes. In particular there is support for exporting certain attributes of a Numpy array as ctypes data-types and there are functions to convert from C arrays to Numpy arrays and back.
For more information, consult the corresponding section in the Numpy Cookbook and the API documentation for numpy.ndarray.ctypes and numpy.ctypeslib.
For the following example, let’s consider a C function in a library that takes an input and an output array, computes the cosine of the input array and stores the result in the output array.
The library consists of the following header file (although this is not strictly needed for this example, we list it for completeness):
void cos_doubles(double * in_array, double * out_array, int size);
The function implementation resides in the following C source file:
#include <math.h>
/* Compute the cosine of each element in in_array, storing the result in
* out_array. */
void cos_doubles(double * in_array, double * out_array, int size){
int i;
for(i=0;i<size;i++){
out_array[i] = cos(in_array[i]);
}
}
And since the library is pure C, we can’t use distutils to compile it, but must use a combination of make and gcc:
m.PHONY : clean
libcos_doubles.so : cos_doubles.o
gcc -shared -Wl,-soname,libcos_doubles.so -o libcos_doubles.so cos_doubles.o
cos_doubles.o : cos_doubles.c
gcc -c -fPIC cos_doubles.c -o cos_doubles.o
clean :
-rm -vf libcos_doubles.so cos_doubles.o cos_doubles.pyc
We can then compile this (on Linux) into the shared library libcos_doubles.so:
$ ls
cos_doubles.c cos_doubles.h cos_doubles.py makefile test_cos_doubles.py
$ make
gcc -c -fPIC cos_doubles.c -o cos_doubles.o
gcc -shared -Wl,-soname,libcos_doubles.so -o libcos_doubles.so cos_doubles.o
$ ls
cos_doubles.c cos_doubles.o libcos_doubles.so* test_cos_doubles.py
cos_doubles.h cos_doubles.py makefile
Now we can proceed to wrap this library via ctypes with direct support for (certain kinds of) Numpy arrays:
""" Example of wrapping a C library function that accepts a C double array as
input using the numpy.ctypeslib. """
import numpy as np
import numpy.ctypeslib as npct
from ctypes import c_int
# input type for the cos_doubles function
# must be a double array, with single dimension that is contiguous
array_1d_double = npct.ndpointer(dtype=np.double, ndim=1, flags='CONTIGUOUS')
# load the library, using numpy mechanisms
libcd = npct.load_library("libcos_doubles", ".")
# setup the return types and argument types
libcd.cos_doubles.restype = None
libcd.cos_doubles.argtypes = [array_1d_double, array_1d_double, c_int]
def cos_doubles_func(in_array, out_array):
return libcd.cos_doubles(in_array, out_array, len(in_array))
And, as before, we convince ourselves that it worked:
import numpy as np
import pylab
import cos_doubles
x = np.arange(0, 2 * np.pi, 0.1)
y = np.empty_like(x)
cos_doubles.cos_doubles_func(x, y)
pylab.plot(x, y)
pylab.show()
SWIG, the Simplified Wrapper Interface Generator, is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages, including Python. The important thing with SWIG is, that it can autogenerate the wrapper code for you. While this is an advantage in terms of development time, it can also be a burden. The generated file tend to be quite large and may not be too human readable and the multiple levels of indirection which are a result of the wrapping process, may be a bit tricky to understand.
Note
The autogenerated C code uses the Python-C-Api.
Advantages
Disadvantages
Let’s imagine that our cos function lives in a cos_module which has been written in c and consists of the source file cos_module.c:
#include <math.h>
double cos_func(double arg){
return cos(arg);
}
and the header file cos_module.h:
double cos_func(double arg);
And our goal is to expose the cos_func to Python. To achieve this with SWIG, we must write an interface file which contains the instructions for SWIG.
/* Example of wrapping cos function from math.h using SWIG. */
%module cos_module
%{
/* the resulting C file should be built as a python extension */
#define SWIG_FILE_WITH_INIT
/* Includes the header in the wrapper code */
#include "cos_module.h"
%}
/* Parse the header file to generate wrappers */
%include "cos_module.h"
As you can see, not too much code is needed here. For this simple example it is enough to simply include the header file in the interface file, to expose the function to Python. However, SWIG does allow for more fine grained inclusion/exclusion of functions found in header files, check the documentation for details.
Generating the compiled wrappers is a two stage process:
from distutils.core import setup, Extension
setup(ext_modules=[Extension("_cos_module",
sources=["cos_module.c", "cos_module.i"])])
$ cd advanced/interfacing_with_c/swig
$ ls
cos_module.c cos_module.h cos_module.i setup.py
$ python setup.py build_ext --inplace
running build_ext
building '_cos_module' extension
swigging cos_module.i to cos_module_wrap.c
swig -python -o cos_module_wrap.c cos_module.i
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/include/python2.7 -c cos_module.c -o build/temp.linux-x86_64-2.7/cos_module.o
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/include/python2.7 -c cos_module_wrap.c -o build/temp.linux-x86_64-2.7/cos_module_wrap.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/cos_module.o build/temp.linux-x86_64-2.7/cos_module_wrap.o -L/home/esc/anaconda/lib -lpython2.7 -o /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/swig/_cos_module.so
$ ls
build/ cos_module.c cos_module.h cos_module.i cos_module.py _cos_module.so* cos_module_wrap.c setup.py
We can now load and execute the cos_module as we have done in the previous examples:
In [1]: import cos_module
In [2]: cos_module?
Type: module
String Form:<module 'cos_module' from 'cos_module.py'>
File: /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/swig/cos_module.py
Docstring: <no docstring>
In [3]: dir(cos_module)
Out[3]:
['__builtins__',
'__doc__',
'__file__',
'__name__',
'__package__',
'_cos_module',
'_newclass',
'_object',
'_swig_getattr',
'_swig_property',
'_swig_repr',
'_swig_setattr',
'_swig_setattr_nondynamic',
'cos_func']
In [4]: cos_module.cos_func(1.0)
Out[4]: 0.5403023058681398
In [5]: cos_module.cos_func(0.0)
Out[5]: 1.0
In [6]: cos_module.cos_func(3.14159265359)
Out[6]: -1.0
Again we test for robustness, and we see that we get a better error message (although, strictly speaking in Python there is no double type):
In [7]: cos_module.cos_func('foo')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-7-11bee483665d> in <module>()
----> 1 cos_module.cos_func('foo')
TypeError: in method 'cos_func', argument 1 of type 'double'
Numpy provides support for SWIG with the numpy.i file. This interface file defines various so-called typemaps which support conversion between Numpy arrays and C-Arrays. In the following example we will take a quick look at how such typemaps work in practice.
We have the same cos_doubles function as in the ctypes example:
void cos_doubles(double * in_array, double * out_array, int size);
#include <math.h>
/* Compute the cosine of each element in in_array, storing the result in
* out_array. */
void cos_doubles(double * in_array, double * out_array, int size){
int i;
for(i=0;i<size;i++){
out_array[i] = cos(in_array[i]);
}
}
This is wrapped as cos_doubles_func using the following SWIG interface file:
/* Example of wrapping a C function that takes a C double array as input using
* numpy typemaps for SWIG. */
%module cos_doubles
%{
/* the resulting C file should be built as a python extension */
#define SWIG_FILE_WITH_INIT
/* Includes the header in the wrapper code */
#include "cos_doubles.h"
%}
/* include the numpy typemaps */
%include "numpy.i"
/* need this for correct module initialization */
%init %{
import_array();
%}
/* typemaps for the two arrays, the second will be modified in-place */
%apply (double* IN_ARRAY1, int DIM1) {(double * in_array, int size_in)}
%apply (double* INPLACE_ARRAY1, int DIM1) {(double * out_array, int size_out)}
/* Wrapper for cos_doubles that massages the types */
%inline %{
/* takes as input two numpy arrays */
void cos_doubles_func(double * in_array, int size_in, double * out_array, int size_out) {
/* calls the original funcion, providing only the size of the first */
cos_doubles(in_array, out_array, size_in);
}
%}
And, as before we can use distutils to wrap this:
from distutils.core import setup, Extension
import numpy
setup(ext_modules=[Extension("_cos_doubles",
sources=["cos_doubles.c", "cos_doubles.i"],
include_dirs=[numpy.get_include()])])
As previously, we need to use include_dirs to specify the location.
$ ls
cos_doubles.c cos_doubles.h cos_doubles.i numpy.i setup.py test_cos_doubles.py
$ python setup.py build_ext -i
running build_ext
building '_cos_doubles' extension
swigging cos_doubles.i to cos_doubles_wrap.c
swig -python -o cos_doubles_wrap.c cos_doubles.i
cos_doubles.i:24: Warning(490): Fragment 'NumPy_Backward_Compatibility' not found.
cos_doubles.i:24: Warning(490): Fragment 'NumPy_Backward_Compatibility' not found.
cos_doubles.i:24: Warning(490): Fragment 'NumPy_Backward_Compatibility' not found.
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include -I/home/esc/anaconda/include/python2.7 -c cos_doubles.c -o build/temp.linux-x86_64-2.7/cos_doubles.o
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include -I/home/esc/anaconda/include/python2.7 -c cos_doubles_wrap.c -o build/temp.linux-x86_64-2.7/cos_doubles_wrap.o
In file included from /home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1722,
from /home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from cos_doubles_wrap.c:2706:
/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
gcc -pthread -shared build/temp.linux-x86_64-2.7/cos_doubles.o build/temp.linux-x86_64-2.7/cos_doubles_wrap.o -L/home/esc/anaconda/lib -lpython2.7 -o /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/swig_numpy/_cos_doubles.so
$ ls
build/ cos_doubles.h cos_doubles.py cos_doubles_wrap.c setup.py
cos_doubles.c cos_doubles.i _cos_doubles.so* numpy.i test_cos_doubles.py
And, as before, we convince ourselves that it worked:
import numpy as np
import pylab
import cos_doubles
x = np.arange(0, 2 * np.pi, 0.1)
y = np.empty_like(x)
cos_doubles.cos_doubles_func(x, y)
pylab.plot(x, y)
pylab.show()
Cython is both a Python-like language for writing C-extensions and an advanced compiler for this language. The Cython language is a superset of Python, which comes with additional constructs that allow you call C functions and annotate variables and class attributes with c types. In this sense one could also call it a Python with types.
In addition to the basic use case of wrapping native code, Cython supports an additional use-case, namely interactive optimization. Basically, one starts out with a pure-Python script and incrementally adds Cython types to the bottleneck code to optimize only those code paths that really matter.
In this sense it is quite similar to SWIG, since the code can be autogenerated but in a sense it also quite similar to ctypes since the wrapping code can (almost) be written in Python.
While others solutions that autogenerate code can be quite difficult to debug (for example SWIG) Cython comes with an extension to the GNU debugger that helps debug Python, Cython and C code.
Note
The autogenerated C code uses the Python-C-Api.
Advantages
Disadvantages
The main Cython code for our cos_module is contained in the file cos_module.pyx:
""" Example of wrapping cos function from math.h using Cython. """
cdef extern from "math.h":
double cos(double arg)
def cos_func(arg):
return cos(arg)
Note the additional keywords such as cdef and extern. Also the cos_func is then pure Python.
Again we can use the standard distutils module, but this time we need some additional pieces from the Cython.Distutils:
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
setup(
cmdclass={'build_ext': build_ext},
ext_modules=[Extension("cos_module", ["cos_module.pyx"])]
)
Compiling this:
$ cd advanced/interfacing_with_c/cython
$ ls
cos_module.pyx setup.py
$ python setup.py build_ext --inplace
running build_ext
cythoning cos_module.pyx to cos_module.c
building 'cos_module' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/include/python2.7 -c cos_module.c -o build/temp.linux-x86_64-2.7/cos_module.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/cos_module.o -L/home/esc/anaconda/lib -lpython2.7 -o /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/cython/cos_module.so
$ ls
build/ cos_module.c cos_module.pyx cos_module.so* setup.py
And running it:
In [1]: import cos_module
In [2]: cos_module?
Type: module
String Form:<module 'cos_module' from 'cos_module.so'>
File: /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/cython/cos_module.so
Docstring: <no docstring>
In [3]: dir(cos_module)
Out[3]:
['__builtins__',
'__doc__',
'__file__',
'__name__',
'__package__',
'__test__',
'cos_func']
In [4]: cos_module.cos_func(1.0)
Out[4]: 0.5403023058681398
In [5]: cos_module.cos_func(0.0)
Out[5]: 1.0
In [6]: cos_module.cos_func(3.14159265359)
Out[6]: -1.0
And, testing a little for robustness, we can see that we get good error messages:
In [7]: cos_module.cos_func('foo')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-7-11bee483665d> in <module>()
----> 1 cos_module.cos_func('foo')
/home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/cython/cos_module.so in cos_module.cos_func (cos_module.c:506)()
TypeError: a float is required
Additionally, it is worth noting that Cython ships with complete declarations for the C math library, which simplifies the code above to become:
""" Simpler example of wrapping cos function from math.h using Cython. """
from libc.math cimport cos
def cos_func(arg):
return cos(arg)
In this case the cimport statement is used to import the cos function.
Cython has support for Numpy via the numpy.pyx file which allows you to add the Numpy array type to your Cython code. I.e. like specifying that variable i is of type int, you can specify that variable a is of type numpy.ndarray with a given dtype. Also, certain optimizations such as bounds checking are supported. Look at the corresponding section in the Cython documentation. In case you want to pass Numpy arrays as C arrays to your Cython wrapped C functions, there is a section about this in the Cython wiki.
In the following example, we will show how to wrap the familiar cos_doubles function using Cython.
void cos_doubles(double * in_array, double * out_array, int size);
#include <math.h>
/* Compute the cosine of each element in in_array, storing the result in
* out_array. */
void cos_doubles(double * in_array, double * out_array, int size){
int i;
for(i=0;i<size;i++){
out_array[i] = cos(in_array[i]);
}
}
This is wrapped as cos_doubles_func using the following Cython code:
""" Example of wrapping a C function that takes C double arrays as input using
the Numpy declarations from Cython """
# cimport the Cython declarations for numpy
cimport numpy as np
# if you want to use the Numpy-C-API from Cython
# (not strictly necessary for this example, but good practice)
np.import_array()
# cdefine the signature of our c function
cdef extern from "cos_doubles.h":
void cos_doubles (double * in_array, double * out_array, int size)
# create the wrapper code, with numpy type annotations
def cos_doubles_func(np.ndarray[double, ndim=1, mode="c"] in_array not None,
np.ndarray[double, ndim=1, mode="c"] out_array not None):
cos_doubles(<double*> np.PyArray_DATA(in_array),
<double*> np.PyArray_DATA(out_array),
in_array.shape[0])
And can be compiled using distutils:
from distutils.core import setup, Extension
import numpy
from Cython.Distutils import build_ext
setup(
cmdclass={'build_ext': build_ext},
ext_modules=[Extension("cos_doubles",
sources=["_cos_doubles.pyx", "cos_doubles.c"],
include_dirs=[numpy.get_include()])],
)
$ ls
cos_doubles.c cos_doubles.h _cos_doubles.pyx setup.py test_cos_doubles.py
$ python setup.py build_ext -i
running build_ext
cythoning _cos_doubles.pyx to _cos_doubles.c
building 'cos_doubles' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include -I/home/esc/anaconda/include/python2.7 -c _cos_doubles.c -o build/temp.linux-x86_64-2.7/_cos_doubles.o
In file included from /home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1722,
from /home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:15,
from _cos_doubles.c:253:
/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/__ufunc_api.h:236: warning: ‘_import_umath’ defined but not used
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/esc/anaconda/lib/python2.7/site-packages/numpy/core/include -I/home/esc/anaconda/include/python2.7 -c cos_doubles.c -o build/temp.linux-x86_64-2.7/cos_doubles.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/_cos_doubles.o build/temp.linux-x86_64-2.7/cos_doubles.o -L/home/esc/anaconda/lib -lpython2.7 -o /home/esc/git-working/scipy-lecture-notes/advanced/interfacing_with_c/cython_numpy/cos_doubles.so
$ ls
build/ _cos_doubles.c cos_doubles.c cos_doubles.h _cos_doubles.pyx cos_doubles.so* setup.py test_cos_doubles.py
And, as before, we convince ourselves that it worked:
import numpy as np
import pylab
import cos_doubles
x = np.arange(0, 2 * np.pi, 0.1)
y = np.empty_like(x)
cos_doubles.cos_doubles_func(x, y)
pylab.plot(x, y)
pylab.show()
In this section four different techniques for interfacing with native code have been presented. The table below roughly summarizes some of the aspects of the techniques.
| x | Part of CPython | Compiled | Autogenerated | Numpy Support |
|---|---|---|---|---|
| Python-C-API | True | True | False | True |
| Ctypes | True | False | False | True |
| Swig | False | True | True | True |
| Cython | False | True | True | True |
Of all three presented techniques, Cython is the most modern and advanced. In particular, the ability to optimize code incrementally by adding types to your Python code is unique.
Since this is a brand new section, the exercises are considered more as pointers as to what to look at next, so pick the ones that you find more interesting. If you have good ideas for exercises, please let us know!