2026-09-05 12:46:42
robots.txt as a stance you pick, not advice you read. Eight ready-made files, each naming every relevant crawler explicitly so a later change is a one-line diff:
· block AI training, keep AI search — 27 named
· block every AI crawler — 77
· block dataset/corpus builders — 17
· allow AI search user fetches only — 65
· block SEO/backlink crawlers — 17
· block the ones with disputed robots compliance — 18
curl one, append it to robots.txt, done.
#robotstxt