🤖
Hosting

Spotting bot traffic in your access log: a practical guide

17.11.2026
← All articles

A large share of requests to any website comes from programs, not people. Some are useful, like search engine crawlers and uptime monitors. Others scrape prices, guess passwords or probe for vulnerable plugins. When a site slows down for no obvious reason or hosting limits are suddenly exhausted, the web server's access.log is the first place to look.

Reading a log line

Most servers use the "combined" format: client IP, time, requested path, status code, response size, referrer and user agent. Here one IP requests page 187 of a catalogue and identifies itself as a bot.

203.0.113.45 - - [12/Oct/2026:14:03:21 +0500] "GET /katalog/?page=187 HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; ExampleBot/2.1)"

In awk, $1 is the IP, $4 the time, $7 the path and $9 the status. The user agent sits inside quotes, so it is easier to split on quotes.

Top IPs and user agents

A normal visitor makes a few dozen requests a day, so tens of thousands from one IP almost always mean automation.

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

Counting by user agent shows which bots announce themselves: search engines, SEO tools, AI crawlers and others.

awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

For a suspicious IP, look at requests per minute. A person opens two or three pages a minute; a bot sends hundreds at a steady rate.

grep '203.0.113.45' access.log | awk '{print substr($4, 2, 17)}' | sort | uniq -c | sort -rn | head

Scanners and password guessers

Malicious bots ask for files that do not exist, so they collect many 404s. Requests for /.env, /phpmyadmin/ or /wp-config.php.bak come from vulnerability scanners, and bursts of POSTs to wp-login.php or xmlrpc.php are brute-force attempts.

awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | head -20
grep -c 'POST /wp-login.php' access.log
grep -E 'POST /(wp-login\.php|xmlrpc\.php)' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Example: a university journal that slowed down every afternoon

A Tashkent university journal on example.uz slowed down for hours each afternoon. The log showed a handful of IPs downloading every format of every archived article in sequence, under an unknown crawler name that ignored robots.txt. After blocking that range and rate-limiting other bots, the site returned to normal and real readers noticed nothing.

Verifying a real Googlebot

Anyone can write "Googlebot" in a user agent. Verify with a two-step DNS check: the reverse record must end in googlebot.com or google.com, and a forward lookup of that name must return the same IP. Other major search engines document similar checks.

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

If either step fails, it is an impostor and blocking it will not affect your search rankings.

Limiting bots: robots.txt, .htaccess and Nginx

Well-behaved bots follow robots.txt, which is the gentlest way to stop unwanted SEO crawlers. Malicious bots ignore it, so they need a server-level block. On Apache shared hosting use .htaccess: the first part rejects by user agent, the second by IP.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (ExampleBot|BadCrawler) [NC]
RewriteRule .* - [F,L]

<RequireAll>
    Require all granted
    Require not ip 203.0.113.45
</RequireAll>

With Nginx on a VPS, rate limiting usually works better than chasing IPs. This allows 5 requests per second per IP with a burst of 20; the excess gets a 503.

# inside the http { ... } block
limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;

# inside server { ... } or a location
location / {
    limit_req zone=perip burst=20 nodelay;
}

Getting logs on shared hosting

On shared hosting, access and error logs are usually available in the ISPmanager logs section or a folder in your account. Download them and run the commands above in a Linux terminal or WSL. If you cannot find them, or bots are exhausting your limits, open a ticket from the sayt.uz cabinet. A weekly look at the top IPs and user agents catches new problems early.

Related articles

💾 Swap on a VPS: do you need it, how much, and how to set it up 🗂️ Working with the ISPmanager File Manager: Uploads, Archives, Editing and Permissions ⏱️ 504 Gateway Timeout: what causes it and how to fix it step by step 📜 Hosting SLA Explained: Uptime Guarantees and Compensation
🌐 Language
🇺🇿 O'zbek 🇺🇿 Ўзбек 🇷🇺 Русский 🇬🇧 English ✓