A large share of requests to any website comes from programs, not people. Some are useful, like search engine crawlers and uptime monitors. Others scrape prices, guess passwords or probe for vulnerable plugins. When a site slows down for no obvious reason or hosting limits are suddenly exhausted, the web server's access.log is the first place to look.
Reading a log line
Most servers use the "combined" format: client IP, time, requested path, status code, response size, referrer and user agent. Here one IP requests page 187 of a catalogue and identifies itself as a bot.
203.0.113.45 - - [12/Oct/2026:14:03:21 +0500] "GET /katalog/?page=187 HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; ExampleBot/2.1)"
In awk, $1 is the IP, $4 the time, $7 the path and $9 the status. The user agent sits inside quotes, so it is easier to split on quotes.
Top IPs and user agents
A normal visitor makes a few dozen requests a day, so tens of thousands from one IP almost always mean automation.
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
Counting by user agent shows which bots announce themselves: search engines, SEO tools, AI crawlers and others.
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
For a suspicious IP, look at requests per minute. A person opens two or three pages a minute; a bot sends hundreds at a steady rate.
grep '203.0.113.45' access.log | awk '{print substr($4, 2, 17)}' | sort | uniq -c | sort -rn | head
Scanners and password guessers
Malicious bots ask for files that do not exist, so they collect many 404s. Requests for /.env, /phpmyadmin/ or /wp-config.php.bak come from vulnerability scanners, and bursts of POSTs to wp-login.php or xmlrpc.php are brute-force attempts.
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | head -20
grep -c 'POST /wp-login.php' access.log
grep -E 'POST /(wp-login\.php|xmlrpc\.php)' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head
Example: a university journal that slowed down every afternoon
A Tashkent university journal on example.uz slowed down for hours each afternoon. The log showed a handful of IPs downloading every format of every archived article in sequence, under an unknown crawler name that ignored robots.txt. After blocking that range and rate-limiting other bots, the site returned to normal and real readers noticed nothing.
Verifying a real Googlebot
Anyone can write "Googlebot" in a user agent. Verify with a two-step DNS check: the reverse record must end in googlebot.com or google.com, and a forward lookup of that name must return the same IP. Other major search engines document similar checks.
host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1
If either step fails, it is an impostor and blocking it will not affect your search rankings.
Limiting bots: robots.txt, .htaccess and Nginx
Well-behaved bots follow robots.txt, which is the gentlest way to stop unwanted SEO crawlers. Malicious bots ignore it, so they need a server-level block. On Apache shared hosting use .htaccess: the first part rejects by user agent, the second by IP.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (ExampleBot|BadCrawler) [NC]
RewriteRule .* - [F,L]
<RequireAll>
Require all granted
Require not ip 203.0.113.45
</RequireAll>
With Nginx on a VPS, rate limiting usually works better than chasing IPs. This allows 5 requests per second per IP with a burst of 20; the excess gets a 503.
# inside the http { ... } block
limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;
# inside server { ... } or a location
location / {
limit_req zone=perip burst=20 nodelay;
}
Getting logs on shared hosting
On shared hosting, access and error logs are usually available in the ISPmanager logs section or a folder in your account. Download them and run the commands above in a Linux terminal or WSL. If you cannot find them, or bots are exhausting your limits, open a ticket from the sayt.uz cabinet. A weekly look at the top IPs and user agents catches new problems early.