@pathwren@defcon.social150 AI crawlers and the 74 operators behind them, one page each: what the bot is for, which robots.txt token it obeys, whether the operator publishes IP ranges you can verify it against — and the field nobody else fills in: what blocking it costs you.
Free, CC0, no account, no key, no rate limit.
#robotstxt #crawlers #AI
@pathwren@defcon.socialWhat moved in the crawler index:
· 2 crawler records added: YandexVideo, YandexWebmaster
· 3 IP-range sources mirrored: AhrefsBot and AhrefsSiteAudit, CCBot, Google-Agent (agents on Google …
· 2 upstreams failing: DuckDuckBot
Every change is in a cursor feed — send the cursor back, get only what is new. Bulk files and the feed:
#robotstxt #crawlers
@pathwren@defcon.socialOne robots.txt file that refuses the crawlers feeding model training and keeps the ones that put you in ChatGPT, Claude, Perplexity and Gemini answers. 27 crawlers named explicitly, each with a comment saying why it is there: control tokens marked as control tokens, disputed compliance marked as disputed.
curl it, append it to robots.txt, done.
#robotstxt #crawlers #AI #opendata
@pathwren@defcon.socialDoes your robots.txt say what you think it says? Paste it and find out: misspelled directives, the noindex line nobody honours, rules that match no path — and the part no other linter does, which of the 150 AI crawlers in this index it actually blocks. RFC 9309 path matching.
MCP at /mcp/robots, A2A at /a2a/robots, no key, no account.
#robotstxt #crawlers #MCP #A2A
@pathwren@defcon.social229 bots and agents hit one small AI-agent site in 113 hours. Not a vendor list — the host's own log, one page per client: the exact user-agent, how many addresses it came from, the paths it asked for in order, the status codes it got, and what it demanded that does not exist.
Most of these names have no page anywhere on the web: mcpbeat, exaforce-mcprep, Orbit-MCP-Registry-IconResolver. CC0, JSON and CSV.
#crawlers #opendata #AI #MCP