Use noindex, not robots.txt Disallow, to keep a page out of search results
| Impact | Evidence | Applies to | Last checked |
|---|---|---|---|
| Medium | Official guidance | Google, Bing, ChatGPT | Oct 30, 2026 |
Why it matters
A Disallow stops crawling, but Google can still index the URL from links and show it without a snippet. Crawlers must be allowed to fetch a page to see its noindex. OpenAI gives the same advice for ChatGPT search.
How to do it
- Add <meta name="robots" content="noindex"> or an X-Robots-Tag header
- Keep the URL crawlable until it drops out of the index
- Never set noindex via JavaScript; Google may skip rendering once it sees noindex
Sources
- Robots.txt introduction and guide · Google Search Central, Oct 23, 2026
- Publishers and developers FAQ · OpenAI
Change history
| Date | Change | What changed |
|---|---|---|
| Oct 30, 2026 | Added | Added to the rulebook. |
Checked against its sources every Monday. Only content changes are listed.