Tootfinder

Opt-in global Mastodon full text search. Join the index!

@shaun@mastodon.xyz
2025-07-09 02:23:07

Got slammed by an unidentified but certainly "#AI"-related #distributed #crawler this week, it drove one site's traffic to 10× average. Today I tired of playing Whac-a-Mole and blocked the two bigge…

Output of a cut, sort, uniq, sort -n job on an Apache format access_log file. It shows around 30K entries per day on July 1, 2, 3, 4. Then suddenly ramping up to 200K and nearly 400K entries on subsequent days. The extra traffic is all from some asshole's "AI" crawler.
Part of an iptables listing from a Linux server. It shows some of my POLICY_DROP_WEB chains which block abusive traffic to 80,443 from various sources. Two rules added today, one for AS136907 (Huawei Cloud) and one for AS45899 (VNPT) have already blocked around 35,000 requests apiece.
@tinoeberl@mastodon.online
2025-08-06 08:11:58

Wie gerade getrötet, ignorieren die KI-Anbieter in ihrem Konkurrenzkampfwahn evtl. die Internetstandards.
Das könnte erklären, warum der Traffic auf meiner Website immer noch stark ansteigt, obwohl ich die #Crawler geblockt habe.
Es könnte natürlich auch an meiner unfassbar hohen Popularität liegen.😁
Hinweis: Ich habe fast alle Artikel nur noch für

@sillon_fictionnel@paperbay.org
2025-06-01 09:26:31

Quand ton site web est archivé par la BnF, c’est que tu as atteint un certain accomplissement.
On râle souvent quand des bots crawlent nos sites, mais quand c’est la BnF, on sourit : c’est pour la postérité, au service de la mémoire collective.
😃
#bnf #crawler

Quand ton site web est archivé par la BnF, c’est que tu as atteint un certain accomplissement.