Tootfinder

What moved in the crawler index:
· Upstream failed: DuckDuckBot (URLError: <urlopen error [Errno 104] Connection reset by peer>)
· 4 IP-range lists refreshed at source: Googlebot, Google special-purpose crawlers, Google user-triggered fetchers and 1 more
· ChatGPT-User: 5 prefixes added, 0 removed
· Upstream recovered: DuckDuckBot
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#AI #opendata #robotstxt

What moved in the crawler index:
· 4 IP-range lists refreshed at source: Googlebot, Google special-purpose crawlers, Google user-triggered fetchers and 1 more
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#AI #opendata #robotstxt

150 AI crawlers and the 74 operators behind them, one page each: what the bot is for, which robots.txt token it obeys, whether the operator publishes IP ranges you can verify it against — and the field nobody else fills in: what blocking it costs you.
Free, CC0, no account, no key, no rate limit.
#robotstxt #crawlers #AI

What moved in the crawler index:
· 2 crawler records added: YandexVideo, YandexWebmaster
· 3 IP-range sources mirrored: AhrefsBot and AhrefsSiteAudit, CCBot, Google-Agent (agents on Google …
· 2 upstreams failing: DuckDuckBot
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#robotstxt #crawlers

robots.txt as a stance you pick, not advice you read. Eight ready-made files, each naming every relevant crawler explicitly so a later change is a one-line diff:
· block AI training, keep AI search — 27 named
· block every AI crawler — 77
· block dataset/corpus builders — 17
· allow AI search user fetches only — 65
· block SEO/backlink crawlers — 17
· block the ones with disputed robots compliance — 18
curl one, append it to robots.txt, done.
#robotstxt

Does your robots.txt say what you think it says? Paste it and find out: misspelled directives, the noindex line nobody honours, rules that match no path — and the part no other linter does, which of the 150 AI crawlers in this index it actually blocks. RFC 9309 path matching.
MCP at /mcp/robots, A2A at /a2a/robots, no key, no account.
#robotstxt #crawlers #MCP #A2A

One robots.txt file that refuses the crawlers feeding model training and keeps the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers. 27 crawlers named explicitly, each with a comment saying why it is there: control tokens marked as control tokens, disputed compliance marked as disputed.
curl it, append it to robots.txt, done.
#robotstxt #crawlers #AI #opendata