How do I allow AI crawlers explicitly?

next-ai-ready build can emit robots.txt with GPTBot, ClaudeBot, and other AI bots allowed, or use app/robots.ts with aiRobots().

Should I advertise llms.txt in robots.txt?

Yes. The build adds comments pointing crawlers to /llms.txt and /llms-full.txt when configured.

guides

Robots.txt

Configure your robots.txt to explicitly allow or block AI crawlers.

next-ai-ready generates a robots.txt that explicitly declares your AI-bot policy. This is important because AI crawlers need clear signals about what they can access.

Default behavior

By default, npx next-ai-ready build writes public/robots.txt that allows all AI bots:

text

User-agent: *
Allow: /

# AI-specific rules
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

# ... (one block per known AI bot)

# AI-readable content
# llms.txt: https://your-site.com/llms.txt
# llms-full.txt: https://your-site.com/llms-full.txt

Each known AI bot gets an explicit Allow: / block. Comments point to your llms.txt and llms-full.txt endpoints.

Configuration

Control robots.txt behavior in ai-ready.config.mjs:

export default defineConfig({
  site: { /* ... */ },
  robots: {
    aiBots: "allow",       // "allow" (default) or "disallow"
    sitemap: true,         // true = auto-generate, string = custom URL
    extra: [
      "Disallow: /admin/",
    ],
  },
})

Field	Type	Default	Description
`aiBots`	`"allow" \	"disallow"`	`"allow"`	Whether AI bots can crawl your site.
`sitemap`	`boolean \	string`	`false`	`true` = auto-generate `Sitemap: <baseUrl>/sitemap.xml`. String = use as-is.
`extra`	`string[]`	`[]`	Raw lines appended verbatim to the output.

Disabling AI bots

To block all AI crawlers:

robots: {
  aiBots: "disallow",
}

This generates Disallow: / for every known AI bot user agent.

Static vs dynamic

By default, robots.txt is written to public/ as a static file. This is the simplest and cheapest approach — it works on any hosting provider.

If you need dynamic robots.txt (e.g. per-request A/B testing), you can:

1. Set emit: { robots: false } in config to skip static generation.

2. Create app/robots.ts using Next.js built-in robots support (this docs site does this).

3. Use aiRobots() from next-ai-ready / @next-ai-ready/core inside app/robots.ts.

Doctor recognizes app/robots.ts + emit.robots: false and will not warn about a missing public/robots.txt.

Known AI bots

The framework detects these user agents:

GPTBot — OpenAI
OAI-SearchBot — OpenAI Search
ChatGPT-User — ChatGPT browsing
PerplexityBot — Perplexity
ClaudeBot — Anthropic
anthropic-ai — Anthropic
Google-Extended — Google AI
CCBot — Common Crawl
Bytespider — ByteDance
Applebot-Extended — Apple