Staging subdomains: hiding a test site from Google the right way

Testing big changes on a separate copy is good practice, usually on a subdomain such as test.example.uz or staging.example.uz. Search engines find these subdomains too: one link, a sitemap or a URL shared through a browser extension is enough. Google then shows two sites with identical text, and customers may land on an unfinished page with old prices or placeholder copy.

Why robots.txt is not enough

Disallow: / in robots.txt stops crawling, not indexing. If something links to the staging URL, Google can still index the address without content. Worse, a crawler that cannot fetch the page never sees its noindex tag, so combining the two works against you. The reliable approach is to let crawlers fetch pages but send a noindex signal, or better, put the site behind a password.

Option one: the X-Robots-Tag header

This header works for HTML, PDFs and images, without touching templates.

# .htaccess (staging folder only!)
<IfModule mod_headers.c>
    Header set X-Robots-Tag "noindex, nofollow"
</IfModule>

# Nginx, staging.example.uz server block
add_header X-Robots-Tag "noindex, nofollow" always;

Verify with curl, and check the main domain as well, because a rule in the wrong folder can remove your real site from search.

curl -sI https://staging.example.uz | grep -i x-robots
curl -sI https://example.uz | grep -i x-robots   # should be empty

Option two: password protection

With HTTP Basic Auth, crawlers get a 401 and cannot index anything, and strangers cannot see unfinished work. Share the credentials only with the people testing the site, and keep the password file outside the web root so it can never be downloaded.

htpasswd -c /home/user/.htpasswd_staging tester

# Apache
AuthType Basic
AuthName "Staging"
AuthUserFile /home/user/.htpasswd_staging
Require valid-user

# Nginx
auth_basic "Staging";
auth_basic_user_file /etc/nginx/.htpasswd_staging;
location /.well-known/acme-challenge/ { auth_basic off; }

The last line matters: Let's Encrypt's HTTP validation does not know your password, so .well-known must stay open or SSL renewal can fail. Webhooks from payment systems or external APIs need similar exceptions.

WordPress settings

WordPress has a "Discourage search engines" checkbox under Settings, Reading. It adds a noindex meta tag and is a useful extra layer, but it is stored in the database, so it travels with the database when you copy staging to production. That is a classic reason sites vanish from search weeks after launch.

If staging is already indexed

Apply one of the methods above, then verify the subdomain as its own property in Google Search Console. The Removals tool hides URLs temporarily; the lasting fix comes when Google recrawls and sees noindex or a 401, which can take from days to weeks depending on how often the subdomain is crawled. Verification needs a DNS TXT record, which you can add in the domain's DNS section of the sayt.uz cabinet.

Example: a school in Namangan

A private school in Namangan spent two months building a redesign on yangi.misol.uz. A teacher shared the link in a parents' chat, and weeks later admission searches began showing the test copy instead of the real site. The developer added a password, removed the test URLs temporarily in Search Console and, on launch day, confirmed with curl that the main site sent neither X-Robots-Tag nor noindex.

Pre-launch checklist

No X-Robots-Tag header on the main domain.
No <meta name="robots" content="noindex"> left in the page source.
WordPress "Discourage search engines" unchecked.
No leftover Disallow: / in robots.txt.
No staging URLs in rel="canonical" links or the sitemap.