AI crawlers are loading your server: how to spot and limit them

Your site suddenly slows down, pages return 503 or 504, and the hosting stats show CPU at its limit. The usual suspects are a traffic spike or an attack, but more and more often the culprit is crawlers run by AI companies and SEO services. They download thousands of pages, and your server pays the bill.

A real case: load jumped fourfold

Recently the load on one of our shared hosting servers rose about four times within minutes. A WordPress site with a downloads section was being hit by a distributed bot calling itself msh-pdfmin/1, fetching around 400 file pages per minute from many IPs. At the same time an online shop was crawled by Meta's meta-externalagent and meta-webindexer at roughly 400 requests per minute across category and filter pages, with MJ12bot and SemrushBot joining in.

Once those user agents were blocked at site level with a 403 in .htaccess, the load fell from about 250 to about 40 within minutes. Real visitors noticed nothing and orders kept coming in.

How to spot bot load

The signs are familiar: slow pages, occasional 503/504 errors and CPU near the limit, while analytics shows no extra visitors, because bots do not run JavaScript. The access log tells the real story. These commands list the busiest user agents and IPs, and the last one counts a specific bot. Scattered IPs with one identical user agent point to a distributed crawler:

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
grep -c "meta-externalagent" access.log

If you cannot reach the logs or the output is hard to read, contact sayt.uz support: our team can identify which bots are loading the server and suggest what to limit.

Search bots versus training bots

Blocking every bot is a mistake. Googlebot and Bingbot index your site and send you visitors. Training bots collect content for language models and give almost nothing back, while SEO crawlers gather data for competitor analysis and leave you only the load.

Google-Extended is not a separate crawler but a robots.txt token that controls whether content may be used for Gemini models; it does not affect search indexing. Applebot-Extended works the same way for Apple.

robots.txt for well-behaved bots

Bots from large companies usually respect robots.txt, so start there. This example shuts out training bots and heavy SEO crawlers while search engines keep indexing the site, just not its heavy pages:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: MJ12bot
User-agent: DotBot
Disallow: /

User-agent: *
Disallow: /*?filter=
Disallow: /*?s=
Disallow: /*add-to-cart=
Crawl-delay: 5

Bing and some SEO bots honour Crawl-delay, but Google ignores it and adjusts its crawl rate based on how your server responds.

.htaccess and nginx for bots that ignore the rules

robots.txt is only a request, and bots like msh-pdfmin/1 never read it. Turn them away at the server instead. On Apache, add this to .htaccess:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (msh-pdfmin|MJ12bot|SemrushBot|DotBot|Bytespider) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (meta-externalagent|meta-webindexer) [NC]
RewriteRule .* - [F,L]

[NC] makes the match case-insensitive and [F] returns 403. A refusal costs far less than building a page, since neither PHP nor the database is touched. On your own VPS with nginx:

if ($http_user_agent ~* (msh-pdfmin|MJ12bot|Bytespider|meta-externalagent)) {
    return 403;
}

Rate limiting and Cloudflare

Some bots only need slowing down. nginx's limit_req caps requests per IP, say two per second, and answers the rest with 429, which also catches bots posing as browsers. Cloudflare offers one-click AI bot blocking, Bot Fight Mode and firewall rules that stop requests before they reach your server.

Protect heavy pages

Often the problem is what bots open, not how many there are. Shop filters create thousands of unique URLs with expensive database queries, and internal search or add-to-cart links are never cached. Close them with Disallow, mark them noindex and add rel="nofollow" to filter links.

Decide bot by bot

Blocking has a price. Shut out OAI-SearchBot, PerplexityBot or Claude-SearchBot and your site may appear less often in ChatGPT and Perplexity answers, where many people now search. A sensible balance is to limit training bots and heavy SEO crawlers while keeping search bots, then watch load and traffic for a couple of days.