Separate public from private routes
Create the sitemap from real public URLs and exclude account, dashboard and parameter combinations that do not represent standalone search intent. Use authentication for private data, not robots.txt alone.
- List the public routes that deserve discovery.
- Set one canonical URL for each route.
- Generate or maintain an accurate XML sitemap.
- Publish robots.txt at the actual domain root.
- Add route-level title, description and social metadata.
- Submit the sitemap in Search Console and inspect representative URLs.
A sitemap is a discovery hint, not a guarantee of indexing. Page quality, internal links and rendering still matter.
Launch checklist
The list below breaks this down into the parts that matter most. Read through each one and note anything that applies to your own setup.
- Sitemap only lists canonical, indexable URLs, not redirects or duplicates.
- Robots.txt references the correct sitemap location.
- Authenticated routes are excluded from the sitemap entirely.
- Canonical tags resolve without a redirect chain.
- Search Console shows no unexpected coverage errors after submission.
Warning signs of crawl waste
Before making changes, work through the points below. Each one covers a different part of the picture, so treat them as a set rather than picking favourites.
- Search Console reporting thousands of crawled pages far above real content volume.
- Parameterised or filtered URLs appearing as separate indexed pages.
- Sitemap containing URLs that return a redirect or error status.