Using third-party Native Libraries

Sometimes, the functionality you need is only available in third-party native libraries. These libraries can still be used from within Pythran, using Pythran’s support for capsules.

Pythran Code

The pythran code requires function pointers to the third-party functions, passed as parameters to your pythran routine, as in the following:

import pythran
%load_ext pythran.magic

%%pythran
#pythran export pythran_cbrt(float64(float64), float64)

def pythran_cbrt(libm_cbrt, val):
    return libm_cbrt(val)

In that case libm_cbrt is expected to be a capsule containing the function pointer to libm’s cbrt (cube root) function.

This capsule can be created using ctypes:

import ctypes

# capsulefactory
PyCapsule_New = ctypes.pythonapi.PyCapsule_New
PyCapsule_New.restype = ctypes.py_object
PyCapsule_New.argtypes = ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p

# load libm
libm = ctypes.CDLL('libm.so.6')

# extract the proper symbol
cbrt = libm.cbrt

# wrap it
cbrt_capsule = PyCapsule_New(cbrt, "double(double)".encode(), None)

The capsule is not usable from Python context (it’s some kind of opaque box) but Pythran knows how to use it. beware, it does not try to do any kind of type verification. It trusts your #pythran export line.

pythran_cbrt(cbrt_capsule, 8.)

2.0

With Pointers

Now, let’s try to use the sincos function. It’s C signature is void sincos(double, double*, double*). How do we pass that to Pythran?

%%pythran

#pythran export pythran_sincos(None(float64, float64*, float64*), float64)
def pythran_sincos(libm_sincos, val):
    import numpy as np
    val_sin, val_cos = np.empty(1), np.empty(1)
    libm_sincos(val, val_sin, val_cos)
    return val_sin[0], val_cos[0]

There is some magic happening here:

• None is used to state the function pointer does not return anything.

• In order to create pointers, we actually create empty one-dimensional array and let pythran handle them as pointer. Beware that you’re in charge of all the memory checking stuff!

Apart from that, we can now call our function with the proper capsule parameter.

sincos_capsule = PyCapsule_New(libm.sincos, "unchecked anyway".encode(), None)

pythran_sincos(sincos_capsule, 0.)

(0.0, 1.0)

With Pythran

It is naturally also possible to use capsule generated by Pythran. In that case, no type shenanigans is required, we’re in our small world.

One just need to use the capsule keyword to indicate we want to generate a capsule.

%%pythran

## This is the capsule.
#pythran export capsule corp((int, str), str set)
def corp(param, lookup):
    res, key = param
    return res if key in lookup else -1

## This is some dummy callsite
#pythran export brief(int, int((int, str), str set)):
def brief(val, capsule):
    return capsule((val, "doctor"), {"some"})

It’s not possible to call the capsule directly, it’s an opaque structure.

try:
    corp((1,"some"),set())
except TypeError as e:
    print(e)

'PyCapsule' object is not callable

It’s possible to pass it to the according pythran function though.

brief(1, corp)

-1

With Cython

The capsule pythran uses may come from Cython-generated code. This uses a little-known feature from cython: api and __pyx_capi__. nogil is of importance here: Pythran releases the GIL, so better not call a cythonized function that uses it.

!find -name 'cube*' -delete

%%file cube.pyx
#cython: language_level=3
cdef api double cube(double x) nogil:
    return x * x * x

Writing cube.pyx

from setuptools import setup
from Cython.Build import cythonize

_ = setup(
    name='cube',
    ext_modules=cythonize("cube.pyx"),
    zip_safe=False,
    # fake CLI call
    script_name='setup.py',
    script_args=['--quiet', 'build_ext', '--inplace']
)

Compiling cube.pyx because it changed.
[1/1] Cythonizing cube.pyx

The cythonized module has a special dictionary that holds the capsule we’re looking for.

import sys
sys.path.insert(0, '.')
import cube
print(type(cube.__pyx_capi__['cube']))

<class 'PyCapsule'>

cython_cube = cube.__pyx_capi__['cube']
pythran_cbrt(cython_cube, 2.)

8.0