Tootfinder

What moved in the crawler index:
· 4 IP-range lists refreshed at source: Googlebot, Google special-purpose crawlers, Google user-triggered fetchers and 1 more
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#AI #opendata #crawlers

What moved in the crawler index:
· 2 crawler records added: YandexVideo, YandexWebmaster
· 3 IP-range sources mirrored: AhrefsBot and AhrefsSiteAudit, CCBot, Google-Agent (agents on Google …
· 2 upstreams failing: DuckDuckBot
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#robotstxt #crawlers

150 AI crawlers and the 74 operators behind them, one page each: what the bot is for, which robots.txt token it obeys, whether the operator publishes IP ranges you can verify it against — and the field nobody else fills in: what blocking it costs you.
Free, CC0, no account, no key, no rate limit.
#robotstxt #crawlers #AI

What moved in the crawler index:
· Upstream recovered: DuckDuckBot
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#AI #opendata #crawlers

The AI Crawler Index is now followable from the Fediverse.
Follow @… and every new entry lands in your timeline: a crawler documented, an operator's published IP ranges refreshed, a bot that quietly changed its user-agent.
Relayed by RSS Parrot from
#RSS #Fediverse #AI #Crawlers

Wie #Statista in der #ZeroClick-Ära überlebt
> #ZeroClick-Antworten kappen … die Verbindung zwischen #Sichtbarkeit und Monetarisierung.
> Anstatt dem Traffic-Verlust tatenlos zuzusehen, wurde [ #Statista ] zum Spielverderber für nicht zahlende #KI- #Crawler. Das Unternehmen traf die … Entscheidung … Inhalte hinter eine #Paywall zu verschieben.
> Die etablierten Erfolgsmetriken wie Sessions, #PageViews oder #Google -#Rankings sind obsolet.
27. Aug. 2026: https://www.computerwoche.de/article/4207372/wie-statista-in-der-zero-click-ara-uberlebt.html