Tootfinder

New on #Quansight PBC blog: Python Wheels: from Tags to Variants
#Python distributions are uniform across different Python versions and platforms. For these distributions, it is sufficient to publish a single wheel that can be installed everywhere. However, some packages are more complex than that; they include compiled Python extensions or binaries. In order to robustly deploy these software on different platforms, you need to publish multiple binary packages, and the installers need to select the one that fits the platform used best.
For a long time, Python wheels made do with a relatively simple mechanism to describe the needed variance: Platform compatibility tags. These tags identified different Python implementations and versions, operating systems, and CPU architectures. Over time, they were extended to facilitate new use cases. To list a couple: PEP 513 added manylinux tags to standardize the core library dependencies on GNU/Linux systems, and PEP 656 added musllinux tags to facilitate Linux systems with musl libc.
However, not all new use cases can be handled effectively within the framework of tags. To list a few:
• The advent of GPU-backed computing made distinguishing different acceleration frameworks such as NVIDIA CUDA or AMD ROCm important.
• As the compatibility with older CPUs became less desirable, many distributions have set baselines for their binary packages to x86-64-v2 microarchitecture level, and Python packages need to be able to express the same requirement.
• Numerical libraries support different BLAS/LAPACK, MPI, OpenMP providers, and wish to enable the users to choose the build matching their desired provider.
While tags could technically be bent to facilitate all these use cases, they would grow quite baroque, and, critically, every change to tags needs to be implemented in all installers and package-related tooling separately, making the adoption difficult.
Facing these limitations, software vendors have employed different solutions to work around the lack of an appropriate mechanism. Eventually, the #WheelNext initiative took up the challenge to design a more robust solution.
"""
#packaging

Another post on #Quansight PBC blog: "BLAS/LAPACK #packaging"
#BLAS and #LAPACK are the standard libraries for linear algebra. The original implementation, often called Netlib LAPACK, developed since the 1980s, nowadays serves primarily as the origin of the standard interface, the reference implementation and a conformance test suite. The end users usually use optimized implementations of the same interfaces. The choice ranges from generically tuned libraries such as OpenBLAS and BLIS, through libraries focused on specific hardware such as Intel® oneMKL, Arm Performance Libraries or the Accelerate framework on macOS, to ATLAS that aims to automatically optimize for a specific system.
The diversity of available libraries, developed in parallel with the standard interfaces, along with vendor-specific extensions and further downstream changes, adds quite a bit of complexity around using these libraries in software, and distributing such software afterwards. This problem entangles implementation authors, consumer software authors, build system maintainers and distribution maintainers. Software authors generally wish to distribute their packages built against a generically optimized BLAS/LAPACK implementation. Advanced users often wish to be able to use a different implementation, more suited to their particular needs. Distributions wish to be able to consistently build software against their system libraries, and ideally provide users the ability to switch between different implementations. Then, build systems need to provide the scaffolding for all of that.
I have recently taken up the work to provide such a scaffolding for the Meson build system; to add support for BLAS and LAPACK dependencies to Meson. While working on it, I had to learn a lot about BLAS/LAPACK packaging: not only how the different implementations differ from one another, but also what is changed by their respective downstream packaging. In this blog post, I would like to organize and share what I have learned.
"""
#CondaForge #Debian #Fedora #Gentoo