noindex, robots.txt and the mistakes that hide a whole site

Most sites that vanish from Google did not get penalised. Someone shipped a staging setting to production, or a robots.txt rule matched more than intended, or the server started answering robots.txt with an error. These mistakes are easy to make, invisible to visitors, and expensive to notice late. This guide explains the two different switches involved and how to check both in a few minutes.

Two different switches: crawling and indexing

robots.txt controls crawling — whether a bot may fetch a URL. noindex controls indexing — whether a fetched page may appear in search results. They are not interchangeable:

You want	Use	Why
A page out of search results	noindex (meta tag or header)	Google must be able to fetch the page to see the instruction
Bots to stop spending requests on a section (admin, cart, internal search)	Disallow in robots.txt	Saves crawling; does not guarantee removal from results
Something truly private	Authentication (login, HTTP auth)	Neither robots.txt nor noindex is access control

A URL blocked in robots.txt can still appear in results — typically as a bare URL with no description — if other pages link to it, because Google knows it exists but cannot read it.

The classic trap: noindex on a page robots.txt blocks

If you add noindex to a page and also disallow it in robots.txt, Google never fetches the page, never sees the noindex, and the URL can stay in the index indefinitely. The fix is counter-intuitive: allow crawling until the page has dropped out, and only then add the Disallow rule if you still want it.

The two forms of noindex

In the HTML <head>:

<meta name="robots" content="noindex">

Or as an HTTP response header, which also works for PDFs and images:

X-Robots-Tag: noindex

The header is the one people forget, because it does not show in "view source". It is often set in the web server or CDN configuration — an nginx add_header X-Robots-Tag "noindex"; copied from a staging server block, or a hosting platform that adds it to preview domains. Check headers directly:

curl -sI https://example.com/ | grep -i x-robots-tag

No output means no header. Any line containing noindex on a page you want ranked is an emergency.

Staging leftovers

The most common way a live site ends up noindexed is a setting that was correct on staging:

• WordPress: Settings → Reading → "Discourage search engines from indexing this site". When ticked, WordPress outputs a robots meta tag with noindex on every page.
• SEO plugins with a global "noindex this site" or per-post-type noindex switch (for example noindex on all pages of a post type that later became your main content).
• Framework environment variables such as a "block indexing" flag that defaults to on outside production and was never set on the new host.
• A robots.txt with Disallow: / deployed together with the rest of the staging files.

Add a one-line check to your launch list and to every migration: fetch the live homepage and its robots.txt from outside your network after DNS switches over.

robots.txt rules that match more than you think

Rules are prefix matches on the path. A few patterns that cause trouble:

User-agent: *
Disallow: /          # blocks the whole site
Disallow: /p         # also blocks /products/, /pricing, /press/
Disallow: /*?        # blocks every URL with a query string
Disallow: /*.php$    # $ anchors the end: blocks /index.php but not /index.php?a=1

• * matches any sequence of characters; $ means "end of URL". They are supported by Google and Bing but were not part of the original standard, so some smaller bots ignore them.
• When several rules match, Google applies the most specific one (the longest matching path); if an Allow and a Disallow are equally specific, Allow wins.
• A bot that finds a group for its own name (User-agent: Googlebot) follows only that group and ignores the * group. Adding a Googlebot group to tweak one rule silently drops every rule you had under * for Googlebot.
• Paths are case-sensitive: Disallow: /Admin does not block /admin.

The DeskSEO checker reports two of these outcomes as critical: robots.txt blocking the whole site, and robots.txt blocking the homepage for Googlebot. The same robots.txt is also evaluated for AI crawlers by the AI bot checker — see AI crawlers and robots.txt for that side.

When robots.txt itself returns an error

How Google treats the robots.txt response matters more than its content in one case:

• 200: rules are applied.
• 404 or other 4xx: treated as "no restrictions" — everything may be crawled. A missing robots.txt is fine.
• 5xx or a timeout: Google treats the whole site as temporarily disallowed and slows or pauses crawling. If the error persists for a long time, Google documents that it may fall back to the last cached version or treat the site as allowed, but you should not rely on that.

So a robots.txt route that crashes, sits behind a firewall challenge, or is served by an application that is down can stop crawling of an otherwise healthy site. Another variant: a framework that answers /robots.txt with the HTML of the homepage and status 200. That parses as garbage, which in practice means no rules — not harmful, but not what you intended. DeskSEO flags robots.txt that is not 200/404, or that returns HTML, as an issue.

How to check a site in five minutes

1. Run the domain through the SEO checker. It fetches robots.txt, evaluates it for Googlebot against the homepage, and reads both the meta robots tag and the X-Robots-Tag header.
2. Open https://your-domain/robots.txt in a private window. Read every Disallow line and ask what else that prefix matches.
3. Run curl -sI on the homepage and on one representative article or product page and look for x-robots-tag.
4. In Google Search Console, use URL Inspection on the same pages. "Excluded by 'noindex' tag" or "Blocked by robots.txt" confirms what Google saw on its last crawl.
5. In the Pages report, look at the trend for "Excluded by 'noindex' tag" and "Blocked by robots.txt". A sudden jump right after a deploy is the clearest sign of a leftover setting.

Recovering after a mistake

Remove the noindex or the blocking rule, confirm with curl that the live response is clean, then request indexing for the most important URLs in URL Inspection and resubmit the sitemap. Recovery is not instant: Google has to recrawl each page, which can take days for important pages and longer for deep ones. Pages that were out of the index briefly usually return to roughly their previous positions, but there is no guaranteed timeline.

Because these mistakes come from deploys rather than from content, the best protection is a check that runs after every change. Scheduled monitoring in a DeskSEO account re-checks the homepage on a timetable and emails when a critical issue such as noindex appears — useful if several people can push to production.