Insights and Experiences of Packaging Python Binary Extensions
2024-07-12 , South Hall 2B

In the domain of scientific and high-performance computing (HPC), software packages are primarily written in compiled languages such as C, C++, and Fortran, complemented by end-user APIs implemented in Python. Such packages frequently incorporate CPU-specific code (e.g. SIMD extensions) and utilise GPU-specific programming models, such as OpenMP and CUDA, to achieve enhanced performance. Despite the recent proliferation of build backends for creating pure Python packages, the distribution of Python packages containing binary extensions poses a unique set of challenges and currently lacks a standardised solution. In this talk, I will share insights and experiences gained from building portable and performant Python wheels for a set of computational neuroscience projects, aiming for compatibility and usage across a diverse of compute platforms, from desktop to large compute clusters.


Expected audience expertise

Intermediate

See also: Presentation slides (8.1 MB)

After several years of being a researcher in cosmology and using Python for studying the large-scale structure of the Universe, I've since moved to the exciting world of high-performance computing, and am currently working on improving software in the computational neuroscience domain at EPFL's Blue Brain Project.