Adventures in compatibility: emulating CPython's C API in PyPy

PyPy is a fast and compliant implementation of Python. In other words, it’s an interpreter for the Python language that can act as a full replacement for the reference interpreter, CPython. It’s optimised to enable efficient just-in-time compilation of Python code to machine code, and has releases matching versions 2.7, 3.5 and soon(ish) 3.6. The PyPy project also developed cffi as a clean and efficient way of interfacing with C code.

However, many libraries in the Python ecosystem are implemented as C extensions, which target CPython’s C API. Many others use Cython, which builds C extensions under the hood. Therefore, PyPy needs an emulation layer for the C API.

This emulation needs to bridge the differences between the implementation languages and the object models of CPython and PyPy. The solution is called cpyext. It’s implemented in a mixture of RPython and C, with most of the API functions and macros implemented in RPython. cpyext exposes PyObjects to the extension code that appear similar to CPython objects (as long as extension writers stay within the fuzzily defined boundaries of the public API) but are merely ‘shadows’ of the real PyPy objects.

After a brief presentation of PyPy, its goals, and its current statuts and roadmap, this talk will dive into the vexed topic of its handling of C extensions. By the end of it, the audience should understand the operating principles of cpyext and have a clearer understanding of what happens when you install and run numpy, for instance, on top of PyPy. Some basic familiarity with CPython internals and how C extensions are made will be assumed.