AI crawler checker and llms.txt generator
Find out whether ChatGPT, Claude, Perplexity, Google, and Bing can reach your site. Mimetic reads your robots.txt and tests each crawler live, then drafts an llms.txt you can publish.
Many sites block AI search without meaning to
A robots.txt rule written to stop AI training can also stop the crawlers that fetch pages for AI answers. Firewall and CDN settings can block them too, even when robots.txt allows them. Either way, the engine can’t read the page it would have cited.
The live test requests your homepage as each crawler and reports the response, so you see what the bots see.
Block training if you want to, keep search open
Search crawlers
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot fetch pages that AI answers cite. Keep these open.
User-request crawlers
ChatGPT-User, Claude-User, and Perplexity-User fetch a page when a person asks about it. Keep these open.
Training crawlers
GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended collect data for model training. Blocking them is a policy choice and doesn't remove you from AI search.
Other free checks
AI visibility checker
How often ChatGPT and Claude name you for your category, plus your AI readiness score.
AI accuracy checker
What ChatGPT and Claude get wrong about your prices, fees, rates, and features, checked against your own site.
Finance claims checker
Rate, fee, and superlative claims a compliance reviewer would question, with suggested rewrites.
Questions clients ask first
Does blocking GPTBot remove me from ChatGPT search?
No, GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which have their own rules in robots.txt.
What is llms.txt?
llms.txt is a plain-text file at the root of a site that summarizes what the company does and links to its most important pages, so AI tools can find them quickly. Support varies by engine, so treat it as a low-cost addition, not a ranking factor.
Why does the live test show a challenge page?
Your firewall or CDN is asking the crawler to prove it is a browser. Most crawlers can't, so they leave without reading the page. Allow verified AI search crawlers in your firewall settings.
Where to go next
Free AI Visibility Checker
Free AI visibility checker. See how often ChatGPT and Claude name your company, your AI readiness score, and your top three fixes.
What AI Gets Wrong About You
Free AI accuracy checker. See which of your prices, fees, rates, and features ChatGPT and Claude get wrong, checked against your own site.
Finance Marketing Claims Checker
Free finance marketing claims checker. Flags rate, fee, APR, and superlative claims a compliance reviewer would question, with suggested rewrites.
See where you stand in AI search
The visibility check is free and returns a score in about a minute. No email needed to see it.
Or write to nic@mimeticgrowth.com