See what a crawler sees
Request a page without a browser and look at the body. If it's little more than an empty root element and a script tag, any crawler that doesn't execute JavaScript sees nothing worth citing.
curl -s https://www.example.com/services | sed -n '/<body/,/<\/body>/p' | head -40Vercel's own research into AI crawler traffic found that the major AI crawlers fetch JavaScript files but don't execute them. Google renders JavaScript, but rendering can be delayed, and other engines vary.
Your options
There are three common ways to give crawlers real HTML. Which suits you depends on how often your pages change.
- Pre-rendering, also called static generation: build each known route to HTML at build time. Best for marketing sites, blogs and guides.
- Server-side rendering: render on each request. Useful for highly dynamic or personalised pages.
- Moving to a framework that does either by default, such as Next.js or Astro, if a larger change is acceptable.
Pre-rendering a Vite React site
For most marketing sites, pre-rendering at build time is the simplest fix. The steps are the same whichever tool you use.
- List every public route, including dynamic ones such as each guide and blog post, from the same data the site uses.
- Render each route to HTML at build time with a pre-rendering tool or a static-generation setup for your router.
- Make sure each route's HTML contains its own title, meta description, canonical and structured data, not just the defaults from index.html.
- Serve a real 404 page with a 404 status for paths that don't exist.
- Keep hydration so the page becomes the full React app once JavaScript loads.
Check the result
Don't rely on how the page looks in a browser. Check what a crawler actually receives.
- curl each key URL and confirm the HTML includes the page's heading, body copy and unique title.
- Confirm robots.txt allows the crawlers you want, such as GPTBot, ClaudeBot and PerplexityBot.
- Make sure the sitemap lists every pre-rendered URL with the canonical domain.
- Request a made-up URL and confirm it returns a 404 status.
Make the content worth citing
Rendering only makes the page readable. To be quoted, a page still needs a direct answer near the top, clear headings, specific facts with sources and structured data that matches what's visible on the page.