---
title: Let AI crawlers in
description: A technical checklist so search engines and AI assistants can reach, read and trust your pages. Run it before any content work.
---

An assistant cannot recommend a page it cannot read. Run this checklist once, then again after any change to your hosting, firewall, routing or layout. Each item takes minutes to check.

## 1. Allow the bots in robots.txt

Open `https://your-domain/robots.txt`. It must not block the bots below. This file allows them and points to your sitemap:

```txt robots.txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
```

`OAI-SearchBot`, `Claude-SearchBot` and `PerplexityBot` feed the search indexes of the assistants. `ChatGPT-User`, `Claude-User` and `Perplexity-User` fetch a page while a user waits. `GPTBot`, `ClaudeBot` and `Google-Extended` control training and grounding. If you block training bots, keep the search and user bots allowed.

## 2. Check your firewall and CDN

A robots.txt that allows a bot does not help if the firewall blocks it first.

- **Cloudflare:** open **AI Crawl Control** and **Security → Bots**. Check that the AI crawlers you want are allowed, and that Bot Fight Mode does not challenge them.
- **Vercel:** open **Firewall**. Check that no bot protection rule or custom rule blocks the AI user agents.
- **Other hosts:** check your WAF and rate limits. A bot that gets `403` or `429` stops coming.

[Count AI bot visits](/docs/ai-bots) to confirm: a bot with zero visits in 30 days is usually blocked.

## 3. Serve the content as HTML

Most AI bots do not run JavaScript. Your text must be in the HTML the server sends.

```bash
curl -s -A "GPTBot" https://example.com/pricing | grep -i "<h1"
```

If the H1 and the main text are missing, turn on server-side rendering or static generation for public pages. Nuxt, Next.js, Astro and SvelteKit all support it.

## 4. Help crawlers find every page

- A `sitemap.xml` lists every public page, and robots.txt names it.
- The sitemap is submitted in Google Search Console and Bing Webmaster Tools.
- Every public page is linked from at least one other page. No orphans.
- Internal links return `200`, with at most one redirect.

## 5. Send one clear signal per page

- One absolute `<link rel="canonical">` per page.
- No `noindex` on pages you want found. Check the meta tags and the `X-Robots-Tag` header.
- One H1 that says what the page is about.
- A unique title and meta description. The title names the product and the page topic.

## 6. Make your facts easy to quote

Assistants repeat facts they can find and trust.

- **A public pricing page** with real numbers. "Contact us" pages are rarely cited for "how much does X cost".
- **An about page** with real people, a company name and a location.
- **Structured data** that matches the visible page: `Organization` site-wide, `SoftwareApplication` or `Product` with `offers` on product pages, `BreadcrumbList` on nested pages, `FAQPage` only where the FAQ is visible.
- **An `llms.txt`** at the root: a short Markdown summary of what you sell, for whom, and links to your key pages.
- **Markdown versions** of key pages (for example `/pricing.md`) for agents that prefer plain text.

## 7. Be fast

Aim for a Largest Contentful Paint under 2.5 seconds and a Cumulative Layout Shift under 0.1 on mobile. Slow pages time out for live fetchers.

:::tip
The [site setup prompts](/docs/prompts/set-up-your-site) ask a coding agent to run this checklist on your repository and fix what it can.
:::