Buyers increasingly ask ChatGPT, Claude, or Perplexity for vendor recommendations instead of scrolling search results. Those assistants can only recommend businesses whose sites their crawlers can actually read. Many company sites block them without knowing it.
Check your robots.txt in 30 seconds
Open yourdomain.com/robots.txt in a browser. Look for these user agents:
GPTBot(OpenAI)ClaudeBot(Anthropic)PerplexityBot(Perplexity)
If you see Disallow: / under any of them, or a blanket User-agent: * with Disallow: /, AI assistants cannot read your site. Some security plugins and CDN “bot protection” presets add these blocks by default.
The fix
Allow the crawlers explicitly:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /If your site runs behind Cloudflare or a similar CDN, also check the bot-management settings: a firewall rule can override anything robots.txt says.
What actually gets you recommended
Crawlability is the entry ticket, not the win. Assistants recommend businesses they can describe confidently, which means pages that answer concrete questions directly, structured data (Organization, Service, FAQ schema), and mentions of your business on third-party sites they trust. A fad file called llms.txt is often pitched for this; measurement across 137,000 sites shows it does nothing. Spend the effort on real content instead.
If you want a second pair of eyes on your site’s AI visibility, describe your setup and I will take a look.