Is your robots.txt shutting AI out?

Plenty of sites add a blanket anti-scraping rule and shut out AI crawlers along with it — then wonder why ChatGPT never mentions them. Enter a URL, see the answer in 10 seconds, and copy the fix.

Your website

No signup · reads robots.txt only · changes nothing

The main AI crawlers

User-agent	Whose	What blocking costs you
`GPTBot`	OpenAI — training and retrieval	ChatGPT has a much harder time getting your content
`OAI-SearchBot`	OpenAI search index	You don't appear in ChatGPT search
`ChatGPT-User`	Fetches when a user opens your link in chat	Even a user deliberately visiting your page fails
`ClaudeBot`	Anthropic	Claude can't reach your content
`PerplexityBot`	Perplexity	Perplexity won't cite you
`Google-Extended`	Google's AI-use control	Harder to appear in Gemini and AI Overviews
`Bytespider`	ByteDance (Doubao)	Doubao can't reach your content
`Baiduspider`	Baidu	Affects Ernie / Baidu AI search

FAQ

How long until a robots.txt change takes effect?

From the crawler's next visit — usually hours to days. It won't put you in AI answers immediately; the content still has to be worth citing.

I don't have a robots.txt. Is that a problem?

No. No file means everything is allowed by default, which is favourable for AI crawling.

If I allow them, aren't they taking my content for free?

A fair concern. The trade-off is that blocking them removes you from the candidate pool almost entirely. The usual answer: allow public marketing content, and disallow paid or proprietary sections specifically.

Why allow ChatGPT-User too?

That's the fetch triggered when a user clicks your link inside a conversation. Blocking it means a person who deliberately wants to read your page can't.

Does this tool change my site?

No. It reads one public file — robots.txt — and hands you a snippet you can paste yourself.

Once you allow them, want to see if they actually came?

This tool gives you a one-time snapshot. A free account keeps the history, re-checks on a schedule, shows the trend over time — and can answer your inquiries automatically. Free forever, no credit card.

Create a free account to save this →

Other free tools

Get WhatsApp APIThree routes + the Oct 2026 change Website GEO auditSee if AI engines can cite you Content citability scoreCheck a draft before you publish Export inquiry fraud checkDecide if a lead is worth the risk Quotation / PI generator10 languages, print straight to PDF Multilingual inquiry repliesEN/ES/DE/FR/AR, fill and send