Tootfinder

Everybody complaining about getting hammered with #AI traffic seems to think that these are crawlers scraping for training data.
How likely is it that this is a complete misconception and this is all inference time?
Most public companies give their cralwers and RAG agents different user agent strings. But what about security services trawling through their data?

#Firecrawl launches /search endpoint for web scraping and data extraction 🔥
🔍 New /search #API endpoint combines web search results with full page content in one call
🤖 Built specifically for #AI

#Firecrawl launches /search endpoint for web scraping and data extraction 🔥
#Firecrawl new /search #API combines web search results with full page content in single call, designed for

There was somebody fussing in my replies to my last link to my blog post about Medium (I don’t see them now; they probably blocked me, but their specific words don’t really matter), and the gist of their message was that they didn’t like that site. On the modern internet, if you have an issue with content written by humans, with no surveillance ads, that doesn’t allow AI scraping or AI slop content, with a business model that makes money… I don’t know how to help you. Honestly.

Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study
Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Emily Wenger
https://arxiv.org/abs/2505.21733

Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study
Online data scraping has taken on new dimensions in recent years, as traditional scrapers have been joined by new AI-specific bots. To counteract unwanted scraping, many sites use tools like the Robots Exclusion Protocol (REP), which places a robots.txt file at the site root to dictate scraper behavior. Yet, the efficacy of the REP is not well-understood. Anecdotal evidence suggests some bots comply poorly with it, but no rigorous study exists to support (or refute) this claim. To understand th…

This morning I null routed another dozen IP addresses for scraping my personal git server using repeated http requests. As per usual, a quick inspection reveals that at least some of them are scraping for LLM data. As always, I have not consented to this use of my non-maintained code, experiments, college coursework, and miscellaneous crap that I for whatever reason decided to self host rather than pushing it to Codeberg.
I mean, if you really want to feed your LLM on a diet that inclu…

Why is BMS software scraping my clipboard? Truly one of the great mysteries of the ages.

Today I fought against AI bots that where scraping / DDoSing one of our customers library OPAC. The quick fix was to block all non-EU IP address. We don't have a good fix yet, but can draw on a lot of ideas that where discussed at #Koha Hackfest. #JohnConner

Interesting. Pointed some test code at google to see if new driver works and got a captcha. Guess they do not want anyone scraping them like they do.