Tootfinder

The /llms.txt file
https://llmstxt.org/

robots.txt (but really /dev/zero)

The release of #Linux 6.17[1] brought a few hours of calmness to the #kernel vanilla repos for #Fedora[2], as that means they just ship three different versions[3]. 😊
Soon the craziness will…

How about having a new /.aibots.txt (or maybe just /.ai.txt)?
All AI-focused crawlers expected to read it.
Start off with the exact same syntax as /.robots.txt - then there’s scope for adding genAI-specific stuff like iP claims and optimized paths and so on.
#genAI

Refreshing to see a Twitter spam account straight up tell you what it is

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions. https://arstechnica.com/tech-policy/2025/09/pay-per-output-ai-firms-blindsided-by-beefed-up-robots-txt-instructions/

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.
“Really Simple Licensing” makes it easier for creators to get paid for AI scraping.

I found this simaltaniousky very informative and extremely obvious in hindsight. It reminded me of when everything seemed a lot more straight forward. I think that I might try this out.
https://www.al3rez.com/todo-txt-journey

Quelle surprise: seriöse Webseiten blocken den Zugang für KI-Trainingszugriff eher als Seiten, deren Zweck der Desinformation dient.
https://arxiv.org/abs/2510.10315

A previously unknown regular satellite of #Uranus has been detected in a series of images obtained by the Near-Infrared Camera (NIRCam) onboard the James Webb Space Telescope on 2025 Feb. 2: http://www.cbat.eps.harvard.edu/iau/cbet/005500/CBET005593.txt -> https://science.nasa.gov/blogs/webb/2025/08/19/new-moon-discovered-orbiting-uranus-using-nasas-webb-telescope/ - the object is located at a projected radial distance of 56250 /- 250 km from Uranus' center in the planet's equatorial plane, initial astrometry is consistent with the moon orbiting on a nearly circular orbit with an orbital period 0.402 days, and the observed IR flux from the object indicates a radius of 4 - 5 km, placing it well below the detection threshold of earlier images from Voyager and the Hubble Space Telescope.

With a FreeBSD pkg repository configuration file set to use quarterly for the one and only repo:
― why is latest (not quarterly) used for bootstrapping?
https://gist.github.com/grahamperrin/1a36d21e9c6d3bc363cee7ecfe779595#file-2…

Any reading recommendations for a small collective looking to move from google dependency to self-hosting?
We're collecting some resources here:
https://doc.patternclub.org/s/QCwRlvO1A#

Hallo @…,
was muß denn in der #robots.txt stehen, um auch im September noch für Euch inidiziert zu werden? #Bing ja wohl nicht mehr, oder doch?

Calendar.txt by Tero Karvinen is a plain text file calendar that's versionable, supports all operating systems, is future-proof, easily syncs with Android, etc: #TextFiles

catherine writes error messages
existence of challenge implies possibility of defeat

Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
Nicolas Steinacker-Olsztyn, Devashish Gosain, Ha Dao
https://arxiv.org/abs/2510.10315 https://

Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web
Large Language Models (LLMs) are increasingly relying on web crawling to stay up to date and accurately answer user queries. These crawlers are expected to honor robots.txt files, which govern automated access. In this study, for the first time, we investigate whether reputable news websites and misinformation sites differ in how they configure these files, particularly in relation to AI crawlers. Analyzing a curated dataset, we find a stark contrast: 60.0% of reputable sites disallow at least …

El intercambio de correos en el que se explica cómo nació el formato de codificación UTF-8 de la mano de sus creadores #utf8

@… @… looks like it found an error in your robots.txt though I don’t see that content in your robots.txt 👀 hmmmmm I wonder if that is a user-agent issue with CloudFlare blocking the lighthouse bot?

As we head into a blizzard of infosec news, stay ahead of the curve by checking out today's Metacurity for the latest developments, including
--Ukraine claims major hack of Russian nuclear submarine,
--SonicWall is aware of flaw exploitation,
--Perplexity is stealthily evading robots.txt,
--FinCen warns of crypt ATM crimes,
--Vietnamese hackers are targeting thousands,
--Informants' data stolen in a Louisiana sheriff's office ransomware attack, …

Ukraine claims major hack of Russian nuclear submarine
SonicWall is aware of flaw exploitation, Perplexity is stealthily evading robots.txt, FinCen warns of crypt ATM crimes, Vietnamese hackers are targeting thousands of victims, Informants' data stolen in a Louisiana sheriff's office ransomware attack, Chanel hit in Salesforce breach, much more

Later, after I keyed n (to not continue), installation did continue.
This is partly understandable, because the y/n prompt was in response to a command that used
-y
Not really a bug, just slightly surprising.
https://gist.github.com/grahamperrin/1…

Z archiwum: jak Jasiu próbował się z kołem przewieźć cugiem z Katowic w okolice Bełsznicy.
http://tek.org.pl/psota-ic.txt
Jak ktoś potrzebuje po warszawsku, to np. tu niżej jest relacja:

Cloudflare says Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocks (Cloudflare)
https://blog.cloudflare.com/perplexity-is-using-stea…

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Alejandro Hern\'andez-Cano, Alexander H\"agele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank \v{D}urech, Ido Hakimi, Juan Garc\'ia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabol\v{c}ec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Ines Altemir Marinas, Mohammad Hossein Amani, Matin An…

Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting robots.txt exclusions and filtering for non-permissive, toxic, and personally identifiabl…

from my link log —
Nontransitive comparison functions lead to out-of-bounds read and write in glibc's qsort().
https://www.qualys.com/2024/01/30/qsort.txt
saved 2025-09-06

#WritersCoffeeClub
19. How do you keep track of dates and events in a WIP?
20. What rôle does death (or undeath!) play in your work?
21. What is your take on the adverb debate?
---
19. Any WIP that needs dates and events is probably already big enough to void memorization.
I went all the way from stickies to txt files to wikis to, eventually, Campfire. I love m…

KIMissbrauch
Cloudflare wirft dem KI-Anbieter ##Perplexity vor, sich mit undeklarierten Crawlern Zugang zu gesperrten Websites zu verschaffen.
Trotz robots.txt-Verboten und IP-Blockaden soll Perplexity mit wechselnden User-Agents und IPs Inhalte verdeckt auslesen.
Das wäre eine Verletzung etablierter Webstandards und Missachtung von Website-Präferenzen.

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

There is an ActivityPub proposal that involves the #DNS.
I have only just discovered it and have not considered it deeply so I am reluctant to make any grand statements. It is not obvious to me why this is useful or better than alternative approaches. It appears to involve the use of TXT RRs, any new de facto use of which makes me skeptical.

[TRACKING] FEP-612d: Identifying ActivityPub Objects through DNS
The proposal has been received. Thank you! This issue tracks discussions and updates to the proposal during the `DRAFT` period. Please post links to relevant discussions as comment to this issue. `dateReceived`: 2023-04-18 If no further actions are taken, the proposal may be set by ed...

»Immer weniger echte Nutzer — Studie zeigt massiven Anstieg von Bot-Traffic im Web:
Menschliche Website-Besucher:innen werden zur Mangelware: Eine neue Analyse zeigt, wie Google, ChatGPT und Co. mit ihren Bots die Spielregeln im Netz verändern - und warum Publisher Alarm schlagen.«
Bots gibt es schon seit den Internet-Anfängen und auch die missachten bewusst die robots.txt Anweisungen.
🫥

Immer weniger echte Nutzer: Studie zeigt massiven Anstieg von Bot-Traffic im Web | t3n
Menschliche Website-Besucher:innen werden zur Mangelware: Eine neue Analyse zeigt, wie Google, ChatGPT und Co. mit ihren Bots die Spielregeln im Netz verändern – und warum Publisher Alarm schlagen. Im zweiten Quartal 2025 ist die Zahl echter Besucher:innen auf Websites um rund zehn Prozent gesunken. Gleichzeitig hat sich der Anteil automatisierter Aufrufe sogar vervierfacht. Während Anfang […]

I'm not sure if this is going to make a difference ( #LLMs weren't able to read #licenses or terms & conditions before when these were not formalized in a "machine-readable" way (plus, besides licenses we already had the robots.txt declarative files; even if those were not as expressive as this new proposal).
So, is this extra work for web developers and maintainers? Are we going to operate under the new assumption that if we didn't do the work of implementing this then we are granting permission to scrapper bots to steal all our online creations?
Or can this be a net gain for creators in some specific way?

Alltså snabb fråga: Jag har fått en .msg fil av en kollega och jag behöver använda adresserna i den för att göra ett utskick till alla på listan. Men jag lyckas inte på något sätt skapa ett nytt mail från filen. Öppnar jag den som txt fil så har den dubbletter av alla namn med ' ' tecken kring varje namn/adress.
Hur i hela världen gör jag för att slippa kopiera namn för namn in i ett nytt mail?

SLEAN: Simple Lightweight Ensemble Analysis Network for Multi-Provider LLM Coordination: Design, Implementation, and Vibe Coding Bug Investigation Case Study
Matheus J. T. Vargas
https://arxiv.org/abs/2510.10010

SLEAN: Simple Lightweight Ensemble Analysis Network for Multi-Provider LLM Coordination: Design, Implementation, and Vibe Coding Bug Investigation Case Study
We present SLEAN (Simple Lightweight Ensemble Analysis Network), a deterministic framework for coordinating multiple LLM providers through text-based prompt orchestration. Unlike complex multi-agent systems requiring specialized infrastructure, SLEAN operates as a simple prompt bridge between LLMs using .txt templates, requiring no deep technical knowledge for deployment. The three-phase protocol formed by independent analysis, cross-critique, and arbitration, filters harmful AI-generated code …

"Today, over two and a half million websites have chosen to completely disallow AI training through our managed robots.txt feature or our managed rule blocking AI Crawlers. Every Cloudflare customer is now able to selectively decide which declared AI crawlers are able to access their content in accordance with their business objectives.
We expected a change in bot and crawler behavior based on these new features, and we expect that the techniques bot operators use to evade detecti…

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

@… two precautions that may help, in lossy situations:
pkg prime-origins | sort -u > /var/tmp/pkg-prime-origins.txt
/usr/local/etc/periodic/daily/411.pkg-backup
If – following an issue – you predict the need to revert to the backup, you can:
service cron stop