Tootfinder

Rzecz jasna, że jest backend #PEP517 dla Sconsa, jednego z najgorszych systemów budowania w dziejach ludzkości.
Rzecz jasna, że nie implementuje poprawnie PEP517.
https:/…

Of course, there's a #PEP517 backend for #SCons, one of the most horrible build systems ever made by humankind.
Of course, it managed not to implement PEP517 correctly.
#Gentoo #Python

New on blog: "the story of #distutils build directory in #Gentoo".
"""
The #Python distutils build system, as well as #setuptools (that it was later merged into), used a two-stage build: first, a build command would prepare a built package version (usually just copy the .py files, sometimes compile Python extensions) into a build directory, then an install command would copy them to the live filesystem, or a staging directory. Curious enough, distutils were an early adopter of out-of-source builds — when used right (which often enough wasn’t the case), no writes would occur in the source directory and all modifications would be done directly in the build directory.
Today, in the #PEP517 era, two-stage builds aren't really relevant anymore. Build systems were turned into black boxes that spew wheels. However, setuptools still internally uses the two-stage build and the build directory, and therefore it still remains relevant to Gentoo eclasses. In this post, I'd like to shortly tell how we dealt with it over the years.
"""
https://blogs.gentoo.org/mgorny/2024/03/13/the-story-of-distutils-build-directory-in-gentoo/

#PEP517 udostępnia ujednolicony interfejs dla wszystkich systemów budowania paczek w Pythonie; tak, by ludzie nie musieli implementować odrębnego kodu dla obsługi każdego z nich.
Właśnie dlatego #Gentoo dla obsługi Pythona ma już specjalną logikę i haki dla 5 różnych systemów budowania opartych o PEP517. …

The state of #Python packaging:
1. Look into `pyproject.toml`. Just random linter configs.
2. Look into `setup.cfg`. More random linter configs.
3. Look into `setup.py`…
[EDIT: 4. It turns out you've missed `requirements.txt`.]
#Gentoo #PEP517

Pomierzyłem sobie, jak poszczególne wersje kodu, obsługującego równoległe budowanie rozszerzeń Pythona (przy pomocy #setuptools) #PEP517, w distutils-r1.eclass #Gentoo, poprawiało się przez lata. Nawet wrzuciłem wy…

I've done some benchmarks to demonstrate how different versions of #Gentoo parallel #Python extension builds (using #setuptools) #PEP517 code in distutils-r1.eclass improved over the years. I've even copied the numbers into a gnumeric spreadsheet. Now I just need to find energy to make a nice chart out of that (probably using PGF/Tikz).
Some numbers for a rough idea:
Cython 3.0.9 (package with a few C extensions):
- serial PEP517 build: 46.7 s
- parallel build / build_ext PEP517: 20.8 s 2.7 s
- parallel PEP517 via DIST_EXTRA_CONFIG: 22.8 s
django 5.0.3 (moderate pure .py package):
- PEP517 build: 5.4 s
- unnecessary build PEP517: 3.1 s 5.3 s
- unnecessary build_ext PEP517: 0.6 s 5.4 s

A good build system is a glorified tool to invoke the toolchain and copy files. The problem with #PEP517 is that it doesn't try to be that. It attempts to be some hybrid between a build system and a package management tool.
It's basically, as if some C project supplied a .spec file (RPM building recipe) instead of a build system, and told you "you can always build a RPM, and then unpack it".
How would a perfect #Python package install process in #Gentoo look like?
1. You unpack a source distribution (.tar.gz).
2. The build system copies .py files from the source distribution into the install directory, using copy-on-write. You end up with N copies of every file (for every Python version) but they'd all share the same data blocks.
3. The package manager copies using CoW (or moves) the files from install directory to the real system. Ideally, again you end up reusing the same data blocks. The install process is efficient, and you save space.
How does it look like?
1. You unpack a source distribution.
2. The build system builds N roughly identical Zip archives, repeatedly recompressing the same source files (note that #GPep517 tries to hack-disable compression, and it usually works).
3. We unpack the Zip archives into the installation directory, getting N disjoint copies of the same .py files…
Well, I suppose we could now use a tool to deduplicate files here. However, think of all the wasted energy: unpack, pack again N times, unpack again N times, find duplicates… Think of all the wasted human effort because someone thought making a package management backend is more enterprisey than a glorified file copying tool.

After getting yet another bug report about #Python package (this time #VirtualEnv) failing to build, because the user did not have new enough #TroveClassifiers installed, I've filed a bug asking #Hatchling to make trove classifier validation either optional or non-fatal.
Right now this simply is not feasible. Technically, every package would need to specify a *minimal* `trove-classifiers` package dependency based on the classifiers they used, and we would have to keep these versions in every #Gentoo ebuild. However, that's unlikely to ever happen, so we'd actually have to check all listed trove classifiers and map them back to package versions. Or, more realistically, just always depend on the newest trove-classifiers available and hope we don't forget to update the dependency.
#PEP517 #packaging

Some #OpenStack projects use #pbr #PEP517 build backend (that uses setuptools internally), while others use #setuptools build backend with pbr plugin. Makes sense.
(No, it doesn't. pbr is just redundant, the backend pointlessly glorifies it and they can't even do it consistently.)
#Gentoo #Python

#PEP517 provides a single interface to all #Python build systems, so people wouldn't have to implement special code to use every one of them.
That's why #Gentoo Python eclasses now carry special logic and hacks for 5 different PEP517 build systems. And I'm only counting detailed stuff, not baseline code like detecting the build backend, handling dependencies and deprecated build backends.
https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass/distutils-r1.eclass?id=75a308e0686ff3757855e05ff06e9ba7418a0458#n1372

There's an open discussion in #Python packaging how `pyproject.toml` and `PKG-INFO` should interact in sdist. Long story short, new version of #Hatchling started taking `PKG-INFO` for granted. For distribution packagers, this means that patching `pyproject.toml` after unpacking sdist suddenly stopped working, and e.g. if you fixed dependency pins in it, hatchling would instead silently used pinned dependencies from `PKG-INFO`.
The discussion is still open, but there's already been some worrying comments, such as people should modify package versions when they patch them…
#Gentoo #PEP517

FYI, it looks like #Hatchling 1.22* has a subtle bug that could result in packages (such as #black) being installed with no entry points. For some reason, the generated sdist archive contains PKG-INFO that doesn't list entry points, and hatchling afterwards uses that to create a wheel with no entry points. It doesn't happen if you go straight for a wheel and there is no PKG-INFO.
#PEP517 #Gentoo

To quote myself:
"""
Honestly, I think the biggest problem here is that #Python #packaging is infinitely complex and counter-intuitive, which means that anyone involved on either end is surprised to find a very high barrier of entry. The #Gentoo Python Guide has already 300 KiB of .rst files, and it is by no means comprehensive. At this point, distribution developers pretty much can't really package anything written in Python without getting a special training and/or senior developer review, and even senior developers have a hard time following the constantly changing landscape.
At the same time, Python framework in Gentoo has already a bunch of safety checks included to detect the most common pitfalls. Again, it is by no means comprehensive and I keep extending it whenever we discover yet another counterintuitive pitfall. This thread makes me think that we will need to add another check to make sure that PKG-INFO is dealt with when pyproject.toml is patched.
"""
#PEP517 #Hatchling

To quote myself:
"""
Honestly, I think the biggest problem here is that #Python #packaging is infinitely complex and counter-intuitive, which means that anyone involved on either end is surprised to find a very high barrier of entry. The #Gentoo Python Guide has already 300 KiB of .rst files, and it is by no means comprehensive. At this point, distribution developers pretty much can't really package anything written in Python without getting a special training and/or senior developer review, and even senior developers have a hard time following the constantly changing landscape.
At the same time, Python framework in Gentoo has already a bunch of safety checks included to detect the most common pitfalls. Again, it is by no means comprehensive and I keep extending it whenever we discover yet another counterintuitive pitfall. This thread makes me think that we will need to add another check to make sure that PKG-INFO is dealt with when pyproject.toml is patched.
"""
#PEP517 #Hatchling