🤖
SEO & marketing

Complete robots.txt guide

10.11.2025
← All articles

robots.txt — tells bots what to index.

Syntax

User-agent: *
Disallow: /admin/
Sitemap: ...

Common use

/admin/, /cabinet/, /api/, /search?

Bots

Googlebot, Yandex, GPTBot.

Errors

Disallow: / on prod — blocks all.

2025

Block AI bots: GPTBot, ClaudeBot.

Related articles

⤵️ Conversion Funnel: Optimizing Every Stage of the Customer Journey 📈 Lifting CTR: getting more clicks from Google search 📊 SEO KPIs: which metrics really matter ❔ People Also Ask: finding and answering questions inside users' heads
🌐 Language
🇺🇿 O⁠'zbek 🇺🇿 Özbek (yangi) 🇺🇿 Ўзбек 🇷🇺 Русский 🇬🇧 English ✓