What Does AI Actually See When It Reads Your Website?

Most AI crawlers retrieve a simplified, text-based version of the page. They can read content contained in the initial HTML response, but they may miss information that requires JavaScript, user interaction, or visual interpretation.

According to Vercel's 2025 crawler analysis, major AI crawlers generally fetch HTML and other page resources without executing JavaScript.^[1]

A website can look complete to a human visitor while appearing nearly empty to an AI system.

How Does an AI Model Read a Web Page?

An AI model does not usually look at your website as a fully rendered visual page.

Instead, a crawler:

1. Requests the page from your server.
2. Receives the page's HTML.
3. Extracts readable text and structured information.
4. Breaks that information into tokens the model can process.

Elements such as colours, spacing, font size, visual grouping, and animations may not survive this process.

Your page structure must therefore make sense without relying on design alone. Headings, lists, paragraphs, captions, links, and semantic HTML help communicate relationships that visual styling would otherwise provide.

Does AI Read HTML or Plain Text?

AI crawlers generally begin with your HTML and then extract the readable text from it.

In Vercel's 2025 analysis, HTML accounted for 57.7% of ChatGPT crawler fetch requests, demonstrating how heavily these systems rely on page markup rather than screenshots of rendered websites.^[1]

Clean HTML gives AI systems a clearer representation of your content. Poorly organized markup, unclear headings, and excessive boilerplate can make the page more difficult to interpret.

What Gets Removed Before AI Reads the Page?

Text extraction commonly removes or reduces:

• Scripts
• Stylesheets
• Decorative elements
• Navigation boilerplate
• Animations
• Interactive controls
• Content added after the initial page load

The remaining output may resemble a simplified reader-mode version of your page.

If an important product description, answer, statistic, or callout exists only inside a JavaScript widget or image, an AI crawler may never receive it as readable text.

Does AI Read Raw HTML or the Rendered Page?

Most dedicated AI crawlers primarily read the raw HTML returned by your server. Rendering JavaScript is generally the exception.

As of Vercel's 2025 analysis, major AI crawlers did not execute JavaScript. Gemini is an important exception because it can rely on Googlebot's crawling and rendering infrastructure.^[1] Google explains that Googlebot processes JavaScript through its Web Rendering Service after the initial crawling stage.^[2]

Important content should be present in the initial HTML response.

Do AI Crawlers Render JavaScript?

Most major AI crawlers do not.

Vercel found that GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, and PerplexityBot could request JavaScript files but did not execute them.^[1]

Content may be invisible to AI when it is:

• Injected through JavaScript
• Retrieved only after an API request
• Displayed after clicking a button
• Loaded through infinite scrolling
• Generated entirely in the browser

Can AI Read React, Vue, or JavaScript Websites?

AI can read React, Vue, Angular, and other JavaScript-based websites — but only when the important content is included in the HTML the crawler receives.

A client-side rendered application may initially return little more than an empty container and references to JavaScript files. The browser then runs those files to build the visible page.

A crawler that does not execute JavaScript sees the original shell rather than the completed application.^[1]

Client-Side vs. Server-Side Rendering

Server-side rendering, static generation, and pre-rendering place readable content in the initial HTML. Client-side rendering requires the visitor's browser to create the content after loading the page.

Google also describes JavaScript rendering as a separate stage that can occur after the initial page crawl.^[2]

Which AI Crawlers Render JavaScript?

Crawler behaviour varies by company, but most dedicated AI crawlers should be treated as non-rendering systems.

Does GPTBot Render JavaScript?

No, not according to Vercel's 2025 crawler data.

OpenAI documents three relevant crawlers:

• GPTBot supports OpenAI's model-development activities.
• OAI-SearchBot supports search functionality in ChatGPT.
• ChatGPT-User retrieves pages in response to user requests.

OpenAI provides separate user-agent information and controls for each crawler.^[3]

GPTBot vs. OAI-SearchBot vs. ChatGPT-User

These crawlers serve different purposes. Blocking GPTBot may limit the use of content for model development, while blocking OAI-SearchBot can affect whether your website appears in ChatGPT search results.

Before editing robots.txt, determine which activity you actually want to restrict.

Does ClaudeBot Render JavaScript?

No, not according to Vercel's 2025 analysis.

Anthropic identifies ClaudeBot, Claude-User, and Claude-SearchBot as separate systems and provides instructions for controlling their access.^[4]

Vercel found that Claude requested JavaScript and image resources but did not execute the JavaScript. Images represented approximately 35% of Claude's requests in the dataset.^[1]

Does PerplexityBot Render JavaScript?

No, not according to Vercel's 2025 crawler logs.

Perplexity documents PerplexityBot and Perplexity-User as its primary crawlers.^[5]

Because Perplexity frequently presents citations alongside its answers, clear and extractable page content can improve the likelihood that your website is understood and used as a source.

Is Google-Extended a Crawler?

No. Google-Extended is a robots.txt control token, not a crawler with its own user-agent.

Google states that Google-Extended does not crawl websites itself. Googlebot performs the crawling, while Google-Extended controls whether crawled content may be used for certain Gemini-related systems.^[6]

What Parts of Your Page Does AI Read?

AI crawlers are most likely to understand information that exists as clear text or structured markup.

What Does AI Usually Read on a Page?

• Page titles
• Meta descriptions
• Headings
• Paragraph text
• Lists
• Link text
• Image alt text
• Captions
• Tables
• Structured data
• Server-rendered accordion content

What Does AI Miss on a Page?

• Client-side rendered text
• Interactive dashboards
• Text embedded inside images
• Canvas elements
• Script-generated charts
• Content loaded after clicking
• Infinite-scroll content
• Login-protected information
• Video or audio without transcripts

Can AI Read Images and Alt Text?

AI crawlers can request image files, but image interpretation varies by platform and context.

Alt text is a more reliable way to explain an image's meaning. Google identifies alt text as an important signal for describing image content.^[8]

Weak alt text: dashboard-image-2

Better alt text: AI visibility dashboard comparing brand citations across ChatGPT, Claude, and Perplexity

Does AI Read Meta Tags and Structured Data?

Yes. Metadata and structured data are included in the page markup, making them accessible without visual rendering.

Structured data can clarify facts such as organization names, product information, authors, publication dates, FAQs, reviews, events, articles, and breadcrumbs.

Google documents JSON-LD, Microdata, and RDFa as supported structured-data formats.^[7]

Can AI Read Accordions and Hidden Content?

It depends on how the content is implemented.

When accordion text is already present in the HTML and merely hidden with CSS, a crawler can usually access it. When the content is fetched or inserted only after a visitor clicks the accordion, a non-rendering crawler may miss it entirely.^[1]

How Can You Check What AI Sees?

Four checks, in increasing order of fidelity: turn JavaScript off, read the raw source, fetch the page the way a crawler does, and compare the extracted text against what a reader sees.

Method 1: Disable JavaScript

Disable JavaScript in your browser and reload the page. Anything that disappears may also be unavailable to non-rendering AI crawlers.

Method 2: View the Page Source

Use your browser's "View Page Source" function and search for an important sentence from the page. Do not rely only on the inspected DOM, because developer tools often show the page after JavaScript has modified it.

Method 3: Fetch the Raw HTML

curl -L https://example.com/page

Search the response for your headings, product information, answers, and calls to action.

Method 4: Compare Extracted Text

Strip the HTML down to readable text and compare it with the visible page. The difference represents a potential AI visibility gap.

Why Is My Content Not Appearing in AI Answers?

The most common reason is that the content is not available in the initial HTML.

Other possible causes include:

• The crawler is blocked in robots.txt.
• The page requires authentication.
• The information is generated through JavaScript.
• The content is buried inside an image or video.
• The page lacks clear headings and direct answers.
• The page is difficult to crawl internally.
• The content is duplicated or unclear.
• The page provides claims without supporting evidence.
• The content does not directly answer likely user questions.

How to Make Your Website Easier for AI to Read

1. Put important content in the initial HTML.
2. Use server-side rendering, static generation, or pre-rendering.
3. Organize pages with descriptive H1, H2, and H3 headings.
4. Answer questions directly beneath the relevant heading.
5. Use short paragraphs and descriptive lists.
6. Add accurate alt text and image captions.
7. Provide transcripts for video and audio.
8. Add appropriate structured data.
9. Use descriptive internal-link anchor text.
10. Test important pages without JavaScript.
11. Keep crawler rules intentional and up to date.
12. Support factual claims with authoritative sources.

Tools such as Detectabli can simplify this process by showing how a page appears from a crawler's perspective, reducing the need to manually fetch, strip, and compare HTML for multiple bots.

Final Takeaway

AI systems do not automatically see everything your website displays.

Most dedicated AI crawlers work from the HTML and readable text returned by your server. When important information depends on JavaScript, visual design, or user interaction, it can disappear from the crawler's version of the page.

The most reliable strategy is to make your website understandable before anything is clicked, rendered, animated, or visually interpreted.

Common Questions

References