Noindex vs disallow: how to keep a page out of Google
Disallow stops crawling, noindex stops indexing, and using both together can backfire. Which to use for private pages, duplicates, PDFs and staging sites.
Published
Tool for this job: Robots.txt Generator and Tester
You add a page to robots.txt to hide it from Google, and weeks later it appears in search results anyway, with the note that no information is available. That is not a bug. Disallow and noindex do different jobs, and mixing them up is one of the most common technical SEO mistakes.
The difference
- Disallow, in robots.txt, tells crawlers not to fetch a URL. It controls crawling.
- Noindex, a meta robots tag in the page or an
X-Robots-TagHTTP header, tells search engines not to show the page in results. It controls indexing.
A disallowed page can still be indexed if other pages link to it: the search engine knows the URL exists, even though it never read the content. In Google Search Console this shows up as “Indexed, though blocked by robots.txt”.
Do not use both on the same page
If you disallow a page and add a noindex tag, the crawler is not allowed to fetch the page, so it never sees the noindex. The page can stay indexed. To remove a page from results, allow crawling and use noindex, and wait until it has dropped out. Only then, if you also want to save crawling, add a disallow.
Which to use
| Situation | Use |
|---|---|
| Thank-you and login pages | noindex |
| Endless filter and search URLs | disallow |
| PDFs you do not want in search | X-Robots-Tag: noindex |
| Staging or test site | password protection |
| Truly private content | authentication |
| Duplicate versions of a page | canonical tag |
Noindex in practice
- In the page head:
<meta name="robots" content="noindex">. - For files without HTML, such as PDFs and images, send the header
X-Robots-Tag: noindexfrom the server. - A noindex line inside robots.txt is not supported by Google, so do not rely on it.
Pages that are already indexed
If an unwanted page is already in Google, adding a disallow rule freezes it there: Google can no longer crawl it to see any change. The order that works is: remove any disallow for that URL, add noindex, wait for Google to recrawl and drop the page (the URL Inspection tool in Search Console shows its status), and only then decide whether to block crawling.
Robots.txt is public
Anyone can read your robots.txt at /robots.txt. Listing /secret-admin/ there advertises it. Robots rules are a polite request to well-behaved crawlers, not access control. Protect anything sensitive with a login.
Test your rules
The Robots.txt Generator and Tester has a Test URLs mode: paste your robots.txt and a list of paths, and it shows which crawler may fetch each one. That catches rules that accidentally block your whole site or important pages. To write a new file, see how to write a robots.txt file.
Removing something urgently
If a page must disappear from Google quickly, for example after publishing personal data by mistake, take it down or protect it, and use the removals tool in Google Search Console, which hides it temporarily while the change is processed.