Can AI crawlers read JavaScript sites? What to check

Many AI crawlers cannot read content that only appears after JavaScript runs. A Vercel and MERJ study found OpenAI's and Anthropic's crawlers, plus PerplexityBot, do not execute JavaScript, while Googlebot, Bingbot and Applebot do render it. Serve key text in the initial HTML with server-side rendering or static generation.

Key takeaways🔗

• Google, Bing and Apple document that their crawlers render JavaScript. OpenAI, Anthropic and Perplexity do not document JavaScript rendering in their crawler docs.

• The most cited test, a December 2024 study by Vercel and MERJ, found that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot did not execute JavaScript.

• If your product names, prices or answers load client-side, those crawlers may see an empty shell.

• Server-side rendering (SSR) or static site generation (SSG) fixes this for most sites. Google itself recommends it because "not all bots can run JavaScript."

• You can test any page in about 10 minutes with view-source, curl and your browser's "disable JavaScript" setting.

Which AI crawlers render JavaScript?🔗

Googlebot, Bingbot and Applebot render JavaScript, according to their owners' documentation. The crawlers behind ChatGPT, Claude and Perplexity did not render it in the largest public test so far, and their vendors' docs do not say they do.

Here is what is documented or tested for each one:

Two honest caveats. First, the Vercel study is from December 2024, and crawlers change. Second, the study does not name Claude-SearchBot, Claude-User or Perplexity-User, so there is no public test result for them. Treat the "no" rows as "assume not until proven otherwise," which is the safe position for anyone selling something.

What did the Vercel and MERJ study actually find?🔗

The Vercel and MERJ study, published December 17, 2024, found that ChatGPT's and Claude's crawlers download JavaScript files but never run them. The researchers analyzed crawler traffic on nextjs.org and the Vercel network, then checked their findings on two job board sites built with different tech stacks (Vercel).

The key numbers from that report:

• JavaScript files made up 11.50% of ChatGPT's crawler requests and 23.84% of Claude's, but the study says "they don't execute them."

• ChatGPT's crawler spent 34.82% of its fetches on 404 pages, and Claude's 34.16%.

• ChatGPT's fetches were 57.70% HTML, while Claude's were 35.17% images.

• In the month measured, GPTBot made 569 million fetches across Vercel's network, Claude 370 million, AppleBot 314 million and PerplexityBot 24.4 million, compared with 4.5 billion for Googlebot.

The study's own top recommendation: "Prioritize server-side rendering for critical content." It says client-side rendering is still fine for "non-essential dynamic elements like view counters, interactive UI enhancements, live chat widgets, and social media feeds," and suggests keeping redirects and sitemaps clean, given how many fetches hit 404s.

Why does JavaScript rendering matter for AI search?🔗

JavaScript rendering matters because an AI crawler that does not run scripts only sees what your server sends in the first HTML response. If that response is an empty <div id="root"></div> plus a script tag, the crawler has nothing to index, quote or cite.

This hits some sites much harder than others:

• Single-page apps built with Create React App, plain Vue or Angular without server rendering often send almost no text in the initial HTML.

• Pricing pages that pull plans from an API after load can hide prices from AI crawlers, which is exactly the detail buyers ask assistants about.

• Product and review widgets loaded by third-party scripts may never be seen.

• Tabs and accordions that fetch content only when clicked are invisible to a crawler that never clicks.

It also matters for Google in a quieter way. Google's own guide says server-side rendering or pre-rendering "is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript" (Google). For more on how ChatGPT search finds pages to begin with, see does ChatGPT use Bing?.

How do you test whether AI crawlers can read your pages?🔗

You test whether AI crawlers can read a page by looking at the raw HTML your server sends, before any JavaScript runs. If your key text is there, crawlers that skip JavaScript can read it. If it is missing, they probably cannot.

Run these four tests on your homepage, pricing page, top product pages and comparison pages:

1. View source. In Chrome, press Ctrl+U (Cmd+Option+U on a Mac) to open the raw HTML. This is not the same as Inspect Element, which shows the page after scripts run. Use Ctrl+F to search for a sentence from the page, a price or a product name.

2. Turn off JavaScript. In Chrome DevTools, open the Command Menu (Ctrl+Shift+P), type "Disable JavaScript" and reload. What you see now is roughly what a non-rendering crawler gets.

3. Fetch it with curl. From a terminal, request the page with an AI crawler's user agent and search the output:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/pricing | grep -i "per month"

The user agent string above is the one OpenAI publishes for GPTBot (OpenAI). This test also catches a second problem: a firewall or CDN that serves AI user agents a block page or a different response.

4. Check the rendered view in Google and Bing. Google Search Console's URL Inspection tool and Bing Webmaster Tools' URL Inspection show how each search engine sees the page. These confirm Google and Bing are fine, but they do not tell you what ChatGPT or Claude see, so do steps 1 to 3 as well.

Score each page simply:

How do you fix a JavaScript site for AI crawlers?🔗

You fix a JavaScript site for AI crawlers by sending the important content in the first HTML response. The cleanest ways are server-side rendering, static site generation or pre-rendering, and most modern frameworks support at least one of them.

Some practical notes:

• Use your framework's server features. Next.js, Nuxt, SvelteKit, Remix and Astro all render on the server or at build time. If you already use one of them, check that key pages are not opted out into client-only rendering.

• Keep the facts in the HTML. Product names, prices, plan limits, locations and FAQ answers should be plain text in the initial response, not injected later.

• Put schema in the server response. JSON-LD added by a tag manager after load may be missed by crawlers that do not run scripts. Our schema markup guide covers which types to use.

• Don't hide answers behind clicks. If a tab loads its content only when opened, render it into the page and use CSS to show and hide it.

• Don't block your scripts in robots.txt. Apple warns that if JavaScript and CSS are blocked, Applebot "may not be able to render the content properly" (Apple), and the same applies to Googlebot and Bingbot.

• Fix 404s and redirects. Given how many AI crawler fetches in the Vercel study hit missing pages, keep your sitemap current and redirect old URLs.

Serving pre-rendered pages only to bots is a stopgap. Google says dynamic rendering "was a workaround and not a long-term solution" and recommends "server-side rendering, static rendering, or hydration" instead (Google).

Does fixing rendering get you recommended by ChatGPT?🔗

Fixing rendering does not on its own get you recommended. It removes a barrier so AI crawlers can read what you have written. Whether ChatGPT, Claude or Perplexity then name your brand depends on your content, your reputation on other sites and your competitors.

That is why I check rendering first and then measure. TrendlyAI runs weekly scans of buyer prompts across ChatGPT, Perplexity, Claude and Google AI Overviews and shows which brands get named. I track TrendlyAI's own domain the same way, and our score is still low, so I know firsthand that readable pages are the floor, not the ceiling.

To see where you stand right now, run the free TrendlyAI check: 5 prompts on ChatGPT and Perplexity, about 30 seconds, no signup. Then work through our GEO audit checklist, or start at the GEO hub.

Frequently asked questions🔗

Does ChatGPT render JavaScript?🔗

OpenAI's crawler documentation does not say its bots render JavaScript, and the Vercel and MERJ study found GPTBot, OAI-SearchBot and ChatGPT-User did not execute it. Assume ChatGPT only sees your initial HTML and put key content there.

Does Claude render JavaScript?🔗

Anthropic's crawler page does not mention JavaScript rendering. In the Vercel and MERJ study, ClaudeBot downloaded JavaScript files but did not run them. Serve important text in the HTML to be safe.

Do Google AI Overviews see JavaScript content?🔗

Google AI Overviews are part of Google Search, which uses Googlebot. Google documents that Googlebot renders pages in headless Chromium, so client-side content can be indexed. Google still recommends server-side or pre-rendering because it is faster and works for other bots.

Is a React site bad for AI search?🔗

No, React is fine when the content is rendered on the server or at build time. The problem is client-only rendering, where the initial HTML is an empty shell. Frameworks such as Next.js render React on the server.

How often should I recheck rendering?🔗

Recheck after any redesign, framework upgrade, CMS change or new CDN setting, and spot-check key pages once a quarter. A single deploy can switch a page to client-side rendering without anyone noticing.

Related reading🔗

• Bing SEO for ChatGPT visibility: a step-by-step guide

• How to get cited by ChatGPT

• Schema markup for AI citations