Skip to content
Sommet
Esc
↑↓navigate↵open⌘Jpreview
On this page

Let AI crawlers in

A technical checklist so search engines and AI assistants can reach, read and trust your pages. Run it before any content work.

An assistant cannot recommend a page it cannot read. Run this checklist once, then again after any change to your hosting, firewall, routing or layout. Each item takes minutes to check.

1. Allow the bots in robots.txt

Open https://your-domain/robots.txt. It must not block the bots below. This file allows them and points to your sitemap:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

OAI-SearchBot, Claude-SearchBot and PerplexityBot feed the search indexes of the assistants. ChatGPT-User, Claude-User and Perplexity-User fetch a page while a user waits. GPTBot, ClaudeBot and Google-Extended control training and grounding. If you block training bots, keep the search and user bots allowed.

2. Check your firewall and CDN

A robots.txt that allows a bot does not help if the firewall blocks it first.

  • Cloudflare: open AI Crawl Control and Security → Bots. Check that the AI crawlers you want are allowed, and that Bot Fight Mode does not challenge them.
  • Vercel: open Firewall. Check that no bot protection rule or custom rule blocks the AI user agents.
  • Other hosts: check your WAF and rate limits. A bot that gets 403 or 429 stops coming.

Count AI bot visits to confirm: a bot with zero visits in 30 days is usually blocked.

3. Serve the content as HTML

Most AI bots do not run JavaScript. Your text must be in the HTML the server sends.

curl -s -A "GPTBot" https://example.com/pricing | grep -i "<h1"

If the H1 and the main text are missing, turn on server-side rendering or static generation for public pages. Nuxt, Next.js, Astro and SvelteKit all support it.

4. Help crawlers find every page

  • A sitemap.xml lists every public page, and robots.txt names it.
  • The sitemap is submitted in Google Search Console and Bing Webmaster Tools.
  • Every public page is linked from at least one other page. No orphans.
  • Internal links return 200, with at most one redirect.

5. Send one clear signal per page

  • One absolute <link rel="canonical"> per page.
  • No noindex on pages you want found. Check the meta tags and the X-Robots-Tag header.
  • One H1 that says what the page is about.
  • A unique title and meta description. The title names the product and the page topic.

6. Make your facts easy to quote

Assistants repeat facts they can find and trust.

  • A public pricing page with real numbers. “Contact us” pages are rarely cited for “how much does X cost”.
  • An about page with real people, a company name and a location.
  • Structured data that matches the visible page: Organization site-wide, SoftwareApplication or Product with offers on product pages, BreadcrumbList on nested pages, FAQPage only where the FAQ is visible.
  • An llms.txt at the root: a short Markdown summary of what you sell, for whom, and links to your key pages.
  • Markdown versions of key pages (for example /pricing.md) for agents that prefer plain text.

7. Be fast

Aim for a Largest Contentful Paint under 2.5 seconds and a Cumulative Layout Shift under 0.1 on mobile. Slow pages time out for live fetchers.

Was this page helpful?