@mgorny@social.treehouse.systemsModern programmers: "oh, let's hijack all #Python package managers in your bashrc without asking for consent, what could possibly go wrong."
And the best joke is, I didn't even really install the package — I was just making a random bugfix and running its test suite in a virtual environment.
#Gentoo #security
@mgorny@pol.socialZawsze cieszy, kiedy autorzy paczki zaakceptują trywialną poprawkę, mimo że oczywisty błąd w kodzie w danej chwili ich nie dotyka.
#Gentoo
@mgorny@social.treehouse.systems
@thesaigoneer@social.linux.pizzaHaving said all that and more: nothing atm comes close to gentoo with dwm. Bar my fav slackware of course 😄
#gentoo #slackware
@mgorny@pol.socialTwoja paczka ma 60 tysięcy testów. Myślisz: "mam dobre pokrycie testami".
Ja myślę: "minie pół godziny, jeden test padnie, i znowu pół godziny testowania…"
#Gentoo
@mgorny@social.treehouse.systems
@portaloffreedom@social.linux.pizza @mgorny@social.treehouse.systemsYour package has 60 thousand tests. You're thinking: "I have good test coverage."
I'm thinking: "half an hour later, a single test will fail. Then I'll spend another half an hour retesting…"
#Gentoo
@mgorny@pol.social @mgorny@social.treehouse.systemsHow you don't want to spend your Saturday: figuring out how many regressions #LLVM developers managed to sneak in just before the 21.x branching, and then bisecting them.
The answer: 2 serious regressions breaking libclc build and install, 2 test regressions in clang and compiler-rt respectively (the former quickly reverted on main, for the second time, but nobody thought of backporting), and 1 assertion failure in experimental backend.
Now I really hope I'll manage a clean run for 21.1.0-rc1 with the biggest zero-day LLVM patchset in the history of #Gentoo.
LLVM downstream maintenance is a nightmare.
@mgorny@pol.social @mgorny@social.treehouse.systemsGiven that #Gentoo #Bugzilla mail is down for three days now, if anybody needed to search for bugs changed since Friday:
https://bugs.gentoo.org/buglist.cgi?f1=delta_ts&o1=greaterthan&order=changeddate&v1=2025-07-11
@thesaigoneer@social.linux.pizzaNerd talk.The satisfaction of having your smb shares mounted via the cli on Gentoo surpasses many things.
#gentoo
@mgorny@pol.social @mgorny@social.treehouse.systemsNowadays in quality #Python: #Gentoo is running #ProtoBuf-related test suite via #PyTest-forked to workaround protobuf segfaulting during GC.
Of course, it implies random programs can segfault on exit too.
https://github.com/protocolbuffers/protobuf/issues/22067
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/protobuf/protobuf-6.31.1.ebuild?id=54e20d4bb0ec99ab868695a2980c4307d179cb10#n150
@mgorny@pol.socialA tak poza tym, to wysłałem parę łatek, by ulepszyć funkcję epytest w #Gentoo.
Wymuszają krótkie podsumowania, załączają tworzenie plików junit .xml, żeby ułatwić maszynowe przetwarzanie wyników, i — co najważniejsze — dodają zmienną EPYTEST_PLUGINS, żeby podawać, które wtyczki mają być załadowane. Będziemy dążyć do tego, by całkiem odejść od domyślnego automatycznego ładowania wtyczek.
@mgorny@social.treehouse.systemsLike other large #FreeSoftware projects, #Gentoo developers have varying degrees of activity. There are some people who dedicate a lot of their free time to Gentoo, maintain hundreds of packages, participate in multiple areas. Then, there are people with narrower interests, lower commit counts, but they are still putting an effort and making Gentoo a better distribution — and that matters. But then, there is the tail.
There is a few of developers whose main talents seem to be 1) finding packages that require absolutely minimal maintenance effort, and 2) justifying their developer status with long essays. I mean, this is getting beyond absurd. It is not just "my packages are all up-to-date". It is not even "my packages require very low maintenance, that's why I'm not doing much". It is literally "I deliberately choose low-maintenance packages, so I don't have to do anything". But of course, all these people definitely need commit access to Gentoo, and show off their Gentoo developer badges, and it's *so damn unfair*.
And in the meantime, other developers are overburdened, and getting burned out. And they step down from more things. And who takes these things over? Of course, not the developers who just admitted to not having much to do…
@mgorny@pol.socialI mamy kolejny komplet jąder #Gentoo #Linux Distribution Kernel (6.1.143, 6.6.96, 6.12.36, 6.15.5). A z nim kilka istotnych zmian:
• Przekopiowałem brakujące zmiany z sys-kernel/gentoo-kernel na sys-kernel/vanilla-kernel — przede wszystkim szersze wsparcie architektur.
• Dodałem domyślną konfigurację …
@mgorny@pol.socialJak bywa to w innych dużych projektach Wolnego Oprogramowania, twórcy #Gentoo miewają różny stopień aktywności. Jedni poświęcają sporą część swojego wolnego czasu, opiekują się setkami paczek, udzielają się w wielu obszarach. Inni mają węższe zainteresowania, produkują się mniej, ale wciąż wkładają wysiłek, by Gentoo było lepsze — i to się liczy. No i jest też ogon.
Mamy więc kilku takich, który…
@mgorny@social.treehouse.systemsSo #Gentoo #Python eclasses are pretty modern, in the sense that they tend to follow the best practices and standards, and eventually deal with deprecations. Nevertheless, they have a long history and carry quite some historical burden, particularly regarding to naming.
The key point is that the eclasses were conceived as a replacement for the old eclasses: "distutils" and "python". Hence, much like we revision ebuilds, I've named the matching eclasses "distutils-r1" and "python-r1". For consistency, I've also used the "-r1" suffix for the remaining eclasses introduced at the time: "python-any-r1", "python-single-r1" and "python-utils-r1" — even though there were never "r0"s.
It didn't take long to realize my first mistake. I've made the multi-impl eclass effectively the "main" eclass, probably largely inspired by the previous Gentoo recommendations. However, in the end I've found out that for the most use cases (i.e. where "distutils-r1" is not involved), there is no real need for multi-impl, and it makes things much harder. So if I were naming them today, I would have named it "python-multi", to indicate the specific use case — and either avoid designating a default at all, or made "python-single" the default.
What aged even worse is the "distutils-r1" eclass. Admittedly, back when it was conceived, distutils was still largely a thing — and there were people (like me) who avoided unnecessary dependency on setuptools. Of course, nowadays it has been entirely devoured by setuptools, and with #PEP517 even "setuptools" wouldn't be a good name anymore. Nowadays, people are getting confused why they are supposed to use "distutils-r1" for, say, Hatchling.
Admittedly, this is something I could have done differently — PEP517 support was a major migration, and involved an explicit switch. Instead of adding DISTUTILS_USE_PEP517 (what a self-contradictory name) variable, I could have forked the eclass. Why didn't I do that? Because there used to be a lot of code shared between the two paths. Of course, over time they diverged more, and eventually I've dropped the legacy support — but the opportunity to rename was lost.
In fact, as a semi-related fact, I've recognized another design problem with the eclass earlier — I should have gone for two eclasses rather than one: a "python-phase" eclass with generic sub-phase support, and a "distutils" (or later "python-pep517") implementing default sub-phases for the common backends. And again, this is precisely how I could have solved the code reuse problem when I introduced PEP517 support.
But then, I didn't anticipate how the eclasses would end up looking like in the end — and I can't really predict what new challenges the Python ecosystem is going to bring us. And I think it's too late to rename or split stuff — too much busywork on everyone.
@mgorny@pol.social @mgorny@social.treehouse.systems @mgorny@pol.socialKiedy poświęcasz godzinę na zaciągnięcie poprawek bezpieczeństwa do wszystkich wersji Pythona w #Gentoo, bo właśnie weszły do repozytorium CPythona i nie było nowych wydań w planach — a kilka godzin później znów poświęcasz czas na zaktualizowanie paczek do nowych wersji, które jednak wydano.
No i oczywiście po drodze dziwisz się, dlaczego nie zamaskowałeś 3.8 poprzednim razem, i drugi raz popełn…
@mgorny@social.treehouse.systemsDays since a random #setuptools project migrated to `pyproject.toml`, removed `MANIFEST.in` and broke source distribution in the process: [0].
#Gentoo
@mgorny@social.treehouse.systemsOne of the goals I've set for further development of #Python eclasses in #Gentoo was to avoid needless complexity. Unfortunately, the subject matter sometimes requires them. However, many of the functions added lately were already manually done in ebuilds for years.
We've started disabling plugin autoloading years ago. First we just did that for individual packages that caused issues. Then, for these where tests ended up being really slow. Finally, pretty much anywhere `python_test()` was declared. Doing it all manually was particularly cumbersome — all I needed for `EPYTEST_PLUGINS` is a good idea how to generalize it.
Similarly, `EPYTEST_XDIST` was added after we have been adding manually `epytest -p xdist -n "$(makeopts_jobs)" --dist=worksteal` — and while at it, I've added `EPYTEST_JOBS` to override the job count.
Perhaps `EPYTEST_TIMEOUT` wasn't that common. However, it was meant to help CI systems that could otherwise get stuck on hanging test.
Similarly, "standard library" version (like `3.9`) matching to `python_gen_cond_dep` was added after a long period of explicitly stating `python3_9 pypy3`. As an extra benefit, this also resolved the problem that at the time `pypy3` could mean different Python versions.
@mgorny@pol.socialBiblioteki eclass związane z Pythonem w #Gentoo są całkiem aktualne. Stosują się do aktualnych zaleceń i standardów, i usuwają na bieżąco rzeczy przestarzałe. Niemniej, mają za sobą długą historię, i najlepiej chyba to widać po nazewnictwie.
Biblioteki te powstały w celu zastąpienia wcześniejszych "distutils" i "python". Dlatego też nazwałem je odpowiednio "distutils-r1&…
@mgorny@social.treehouse.systemsSo I've just bumped a bunch of old #Gentoo packages to EAPI 8. Some of them haven't been updated for 6 years. And do you know what's best? They still worked — their build systems work, they compile and they just work. Unlike most of the stuff developed these days.
#autotools #C
@mgorny@pol.socialWłaśnie zaktualizowałem trochę starych paczek #Gentoo do EAPI 8. Niektóre nie były aktualizowane od 6 lat. I wiecie, co jest najlepsze? Że nadal działają — systemy budowania działają, kod się kompiluje, programy działają. W odróżnieniu od większości nowego oprogramowania.
#autotools
@mgorny@social.treehouse.systems @mgorny@pol.social @mgorny@pol.socialNieudany poranek z nowymi wersjami paczek Pythona dla #Gentoo:
1. Projekt, który zwlekał z wydaniem nowej wersji z poprawkami bezpieczeństwa 4 lata, w końcu wydał nową wersję. Oczywiście, jak się robi jedno wydanie na 7 lat, to definitywnie trzeba w tym czasie zmienić system budowania na zepsutą hybrydę #PythonPoetry
@mgorny@pol.socialZdradzę wam sekret: nazwa "portage" pochodzi od pora — tego warzywa.
#Gentoo
@mgorny@social.treehouse.systemsNew set of #Gentoo #Linux Distribution Kernels (6.1.143, 6.6.96, 6.12.36, 6.15.5) is out. This set brings some major changes:
• I've backported a bunch of changes from sys-kernel/gentoo-kernel to sys-kernel/vanilla-kernel that were missing — notably wider architecture support.
• I've added default #RISCV configs to 6.12 (in addition to 6.15), since Fedora had them.
• All three packages are based off the baseline kernel tarball upstream patch (vanilla-kernel used to fetch patch-level tarball every time, and gentoo-kernel* used genpatches for patch versions). This should reduce disk space and bandwidth use.
• All three packages now support verify-sig. Rather than verifying the uncompressed tarball signature, we now use upstream `sha256sums.asc` file to verify the compressed tarball and patch.
• sys-kernel/gentoo-kernel* now repackages genpatches. This means patchset that's much leaner and faster to apply (since we just fetch and use the combined upstream patch rather than including point patches). This also means that we are able to release Distribution Kernels before gentoo-sources are done.
The changes still need to be done to 5.15 and 5.10 branches — we're going to do for the next upstream releases of these.
#kernel
@mgorny@pol.socialPaczka Pythona, której nie idzie zainstalować na Pythonie 3.14, bo autor koniecznie musiał zaimplementować przetwarzanie AST na ponad 200 linii w `setup.py`? Dlaczego nie.
#Gentoo #Python #setuptools
@mgorny@pol.socialPaczki Pythona:
"A pamiętacie tę całkowicie przypadkową wtyczkę PyTesta, która nie jest rozwijana od 2018 roku, i którą musieliście dodać do #Gentoo, bo postanowiliśmy jej używać bez jakiegokolwiek powodu? No cóż, właśnie przestaliśmy. A tak przy okazji — właśnie udało nam się znaleźć kolejną wtyczkę, która po raz trzeci wynajduje na nowo obsługę niestabilnych testów. Miłej zabawy!"
@mgorny@social.treehouse.systemsA while ago, I've followed the example given by #Fedora and unbundled ensurepip wheels from #Python in #Gentoo (just checked — "a while ago" was 3 years ago). This had the important advantage that it enabled us to update these wheels along with the actual pip and setuptools packages, meaning new virtual environments would get fresh versions rather than whatever CPython happened to bundle at the time of release.
I had considered using our system packages to prepare these wheels, but since we were already unbundling dependencies back then, that couldn't work. So I just went with fetching upstream wheels from PyPI. Why not build them from source instead? Well, besides feeling unnecessary (it's not like the PyPI wheels are actually binary packages), we probably didn't have the right kind of eclass support for that at the time.
Inspired by @…, today I've tried preparing new revisions of ensurepip packages that actually do build everything from source. So what changed, and why should building from source matter now? Firstly, as part of the wheel reuse patches, we do have a reasonably clean architecture to grab the wheels created as part of the PEP517 build. Secondly, since we're unbundling dependencies from pip and setuptools, we're effectively testing different packages than these installed as ensurepip wheels — and so it would be meaningful to test both variants. Thirdly, building from source is going to make patching easier, and at the very least enable user patching.
While at it, I've refreshed the test suite runs in all three regular packages (pip, setuptools and wheel — we need an "ensurepip" wheel for the last because of test suites). And of course, I hit some test failures in testing the versions with bundled dependencies, and I've discovered a random bug in #PyPy.
https://github.com/gentoo/gentoo/pull/42882 (yes, we haven't moved yet)
https://github.com/pypy/pypy/issues/5306
@mgorny@social.treehouse.systemsIn other news, I've sent a few fun patches to improve epytest in #Gentoo.
This includes forcing short summaries, creating junit .xml for machine processing, and most importantly, EPYTEST_PLUGINS to handle specifying the plugins to load. The goal is to eventually move away from plugin autoloading by default.
#PyTest #Python
@mgorny@social.treehouse.systemsWhen you spend an hour backporting #CPython #security fixes to all versions of #Python #Gentoo, because there was no planned security release, and a few hours later you spend time again bumping to the unexpected security releases.
And then you are surprised why you didn't mask Python 3.8 yet, and repeat the same mistake.
Oh, and ofc update your CPython and PyPy (fixed PyPy only in Gentoo).
@mgorny@social.treehouse.systemsThe time has come for me to propose that #Gentoo leaves #GitHub.
#Microsoft has been purveying #enshittification of this platform for years now. Pushing #Copilot everywhere was the last straw.
#AI #LLM
@mgorny@social.treehouse.systemsNew reason not to use #PythonPoetry just dropped: they reinvented "reproducible builds", poorly. The problem is, they missed the purpose of reproducible builds entirely and they use it for source distributions too, and when you don't use SOURCE_DATE_EPOCH, they force all files to epoch (as in timestamp 0) instead of leaving them alone.
Like, all source distributions created by Poetry and uploaded to #PyPI now have 1970 timestamps that, simply speaking, break stuff. The most absurd thing is that ZIP can't handle that timestamp, so they override it and use another date for wheels 🤦.
#Gentoo #PEP517
@mgorny@social.treehouse.systemsA #Python package that can't be installed on Python 3.14, because the author had to implement a 200 line custom AST parser in `setup.py`? Yeah, why not.
#Gentoo #packaging #setuptools
@mgorny@social.treehouse.systemsA bad #Python bump morning in #Gentoo:
1. A project that couldn't be bothered to make a release with a security fix for 4 years finally made a release. Of course, if you make one release in 7 years, it is definitely a good idea to replace your build system with a broken #PythonPoetry #setuptools hybrid.
2. Another project made a release with a bunch of test failures — that were fixed in "master" branch already at the time, but I guess nobody bothered testing the release branch.
3. Just discovered that a bunch of projects are using pkg_resources namespaces again — and we were supposed to have gotten rid of them years ago! Of course it's #Google. And on top of that, since pkg_resources are now throwing deprecation warnings, they are indirectly breaking random other test suites.
On the positive side, test_lolwut is failing for me in redis-py.
@mgorny@social.treehouse.systems#Python #packaging be like:
"Remember the totally random #PyTest plugin that died in 2018, that we forced you to add to #Gentoo, because we decided to start using it for no good reason? Well, we just stopped. Also, we just found a #NIH plugin that reinvents flaky test handling for the third time, enjoy!"
(Fortunately, it's compatible enough with pytest-rerunfailures, so we can ignore it.)
@mgorny@social.treehouse.systems#PyYAML rejected #freethreading support. As a result, a new fork has been created with freethreading support. Given the fork's focus on freethreading, it supports only Python 3.13 . Given the lack of environment markers for freethreading (yet), packages end up depending on PyYAML-ft for >=3.13 (including non-freethreading builds), and PyYAML for <3.13.
Isn't #Python #packaging great?
#Gentoo