Pick the right control
Before making changes, work through the points below. Each one covers a different part of the picture, so treat them as a set rather than picking favourites.
- Robots.txt: a crawl guidance file at the domain root.
- Noindex: a page-level instruction not to include a URL in results.
- Authentication: the real control for private content.
- Canonical: a preference for the primary version of duplicate or similar pages.
With the right control chosen, apply it using the sequence below.
- List public pages that should be discovered.
- Identify private, duplicate or low-value routes.
- Apply noindex to pages that should exist but not appear in search.
- Check robots.txt does not block important CSS, JS or public paths.
- Test the published URL and inspect the resulting directives.
Do not disallow a URL in robots.txt and expect search engines to see its noindex tag. If crawlers cannot fetch the page, they may not see the instruction.
Diagnostic sequence for indexing problems
Work through these steps in order. Each one builds on the last, so resist skipping ahead even if a step feels familiar.
- Check whether the URL is disallowed in robots.txt.
- Fetch the page and confirm no noindex tag is present unintentionally.
- Confirm the canonical points to itself, not another URL.
- Check the sitemap includes the page if it should be indexed.
- Request indexing in Search Console once the above are confirmed correct.
Pages that usually need noindex, not robots.txt
A few moving parts are involved here. The points below cover each of them, so you can see the full scope before deciding where to start.
- Thank you and confirmation pages.
- Internal search result pages.
- Duplicate filtered or sorted variants of a listing.
- Staging or preview subdomains left publicly reachable.