Tootfinder

from my link log —
ai.robots.txt: A list of AI agents and robots to block.
https://github.com/ai-robots-txt/ai.robots.txt
saved 2025-01-16

GitHub - ai-robots-txt/ai.robots.txt: A list of AI agents and robots to block.
A list of AI agents and robots to block. Contribute to ai-robots-txt/ai.robots.txt development by creating an account on GitHub.

Writing code that ignores robots.txt is a professional ethics violation.
This is a toot about #AI

Last week, our students learned how to conduct a proper evaluation for an NLP experiment. To this end, we introduced a small textcorpus with sentences about Joseph Fourier, who counts as one of the discoverers of the greenhouse effect, responsible for global warming.

Is there anything I can put in robots.txt that will stop Scrapy?
Failing that, let’s take the ship up and nuke the site from orbit. It’s the only way to be sure.

#AI #honeypots huh

The /llms.txt file – llms-txt
A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

#AI #honeypots huh

The /llms.txt file – llms-txt
A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

Fsck GMail!
@ IN TXT "v=spf1 all"

I just discovered TXT... feels all kinds of uplifting for a Monday morning :)
#TomorrowXTogether #KPop

From HOSTS.TXT to Modern Internet Infrastructure
🌐 #hoststxt

From HOSTS.TXT to Modern Internet Infrastructure | AXON Shield
The development of DNS demonstrates an impressive journey from its initial basic form into a modern distributed system which provides high resilience. The internet initially used a basic centralized text file named HOSTS.TXT for its operations. The rapid internet expansion made the initial text file system unworkable so developers created a new solution which could scale dynamically. The system evolves because organizations need better scalability and absolute reliability along with robust secu…

There is now also a CBET about the new interstellar #comet 3I/ATLAS: http://www.cbat.eps.harvard.edu/iau/cbet/005500/CBET005578.txt - it comes with an even more precise orbit based on astrometry back to 5 June and predicts 13th magnitude with 60° elongation after perihelion in November. The current magnitude is about 17.7.

Control How Your Content Is Used for AI Training With Cloudflare (Cloudflare Blog, 1 July 2024)
#MediaLit

Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf, and allowing targeted blocking of AI crawling on sites that serve ads.

From the digital archives: #AWS #EC2 IP ranges from 14 years ago.
https://

🥱
The day SHALL start.
Regards not given,
RFC2119
https://ietf.org/rfc/rfc2119.txt

To whomever praises #Claude #LLM:
ClaudeBot has made 20k requests to bugs.gentoo.org today. 15k of them were repeatedly fetching robots.txt. That surely is a sign of great code quality.
#AI

Jak ktoś chwali sobie #Claude #LLM, to wspomnę:
ClaudeBot dziś wykonał 20 tysięcy żądań do bugs.gentoo.org. Spośród nich, 15 tysięcy w kółko ciągnęło plik robots.txt. Zaprawdę wysokiej jakości kod.
#AI

Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study
Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Emily Wenger
https://arxiv.org/abs/2505.21733

Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study
Online data scraping has taken on new dimensions in recent years, as traditional scrapers have been joined by new AI-specific bots. To counteract unwanted scraping, many sites use tools like the Robots Exclusion Protocol (REP), which places a robots.txt file at the site root to dictate scraper behavior. Yet, the efficacy of the REP is not well-understood. Anecdotal evidence suggests some bots comply poorly with it, but no rigorous study exists to support (or refute) this claim. To understand th…

new cool idea: whenever anything requests /llms.txt or *.md serve them 42.zip
#ai #noai

new cool idea: whenever anything requests /llms.txt or *.md serve them 42.zip
#ai #noai

📝🗃️ 𝗿𝗱𝗼𝗰𝗱𝘂𝗺𝗽: Dump ‘R’ Package Source, Documentation, and Vignettes into One File for use in LLMs #rstats #LLM is on CRAN https://www.ekotov.pro/rdocdum…

rdocdump: Dump R Package Documentation and Vignettes into One File
Dump R Package Documentation and Vignettes into One File

『DOSの人が困るので、ファイル名は8文字のアルファベット大文字と _ と数字の組みあわせ（8.3形式）でお願いします』 -- README~1.TXT

Been designing distributed counters for NATS. Pretty happy with this.
50k/second unoptimised and on a single counter - but we will support aggregation of regional to global etc.
Hard dist sys problems made trivial to use and operate 💪💪
https://gist.github.com/ripienaar/d95d