robots.txt — tells bots what to index.
Syntax
User-agent: *
Disallow: /admin/
Sitemap: ...
Common use
/admin/, /cabinet/, /api/, /search?
Bots
Googlebot, Yandex, GPTBot.
Errors
Disallow: / on prod — blocks all.
2025
Block AI bots: GPTBot, ClaudeBot.
robots.txt — tells bots what to index.
User-agent: *
Disallow: /admin/
Sitemap: ...
/admin/, /cabinet/, /api/, /search?
Googlebot, Yandex, GPTBot.
Disallow: / on prod — blocks all.
Block AI bots: GPTBot, ClaudeBot.