EVOLVXAI
AI Marketing

llms.txt, robots.txt and ChatGPT: What Your UAE Website Actually Needs (2026)

·5 min read·Sawan Kumar

To appear in ChatGPT search, your website must allow OpenAI's OAI-SearchBot in robots.txt and in any host or CDN security layer. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that any public website can appear, with placement not guaranteed (OpenAI crawler documentation). llms.txt is optional. It is a community proposal, Google says its search ignores the file, and we found no statement in OpenAI's, Anthropic's or Perplexity's crawler documentation that their crawlers read it. Fix robots.txt first; treat llms.txt as a 15-minute extra at most.

Which AI Bots Should Your Website Allow?

CompanyBotWhat the company says it is forIf you block it
OpenAIOAI-SearchBotSurfacing websites in ChatGPT's search featuresNot shown in ChatGPT search answers
OpenAIChatGPT-UserUser-initiated actions, not automatic crawlingDoes not decide search eligibility; that is controlled by OAI-SearchBot
OpenAIGPTBotMaking OpenAI's generative AI foundation models more useful and safeOpts out of training; independent of search
PerplexityPerplexityBotSurfacing and linking websites in Perplexity search resultsPerplexity recommends allowing it to appear in search results
PerplexityPerplexity-UserSupporting user actions; not used for crawling or trainingSeparate from PerplexityBot
AnthropicClaude-SearchBotIndexing content to improve search result qualityMay reduce visibility and accuracy in user search results
AnthropicClaude-UserUser-directed requestsMay reduce visibility in user-directed web search
AnthropicClaudeBotCollecting web content that may contribute to trainingOpts out of training
GoogleGooglebotGoogle Search, including AI Overviews and AI ModePages cannot be indexed, so they are not eligible for AI features

Sources: OpenAI, Perplexity, Anthropic, Google. OpenAI says each of its settings is independent of the others, so you can allow search and still block training.

What Does a Working robots.txt Look Like?

This version allows everything. Replace the sitemap line with your own.

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Sitemap: https://yourwebsite.com/sitemap.xml

To stay in search but opt out of training, add these two blocks. OpenAI and Anthropic both say their bots follow robots.txt.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Anthropic notes that rules must be set for each subdomain separately, and that blocking IP addresses may not reliably opt a site out.

Is Your Host or CDN Blocking the Bots?

robots.txt is only half of access. Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as its first AI-search best practice (Google Search Central). Some website builders and security services block unknown bots by default. Ask your web host or CDN provider whether they do, and ask them to allow OpenAI's search bot. OpenAI publishes the IP addresses its bots use.

After you change anything, wait. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.

What Is llms.txt, and Should You Publish One?

llms.txt is a proposal by Jeremy Howard, first published on 3 September 2024. Its own page calls it "A proposal to standardise on using an /llms.txt file," open for community input, not a formal standard (llmstxt.org). It is a Markdown file with:

  • one H1 with the site name (the only required part)
  • an optional short summary in a blockquote
  • H2 sections with lists of links, each as [name](url) plus an optional note

Does it work? Google says no: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and llms.txt files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them" (Google Search Central). For OpenAI, Anthropic and Perplexity, we found crawler documentation covering robots.txt but no statement that their crawlers read llms.txt. The proposal says the file is meant for agents to read, but that is the proposal's intent, not a platform commitment.

Our view: it is harmless, so publish one if it takes you 15 minutes. EvolvXAI publishes one for this blog. Do not pay a service for it, and do not let it replace the work that Google and OpenAI do document: crawler access, a complete Business Profile, and key facts as text.

Example for a restaurant:

# Example Restaurant

> Family restaurant serving North Indian food in Al Barsha, Dubai. Halal. Parking available.

## Key pages
- [Menu](https://yourwebsite.com/menu): Dishes and prices
- [Opening hours and location](https://yourwebsite.com/contact): Address, hours, parking
- [Book a table](https://yourwebsite.com/book): Reservations

How Do You Test That It Worked?

  1. Open yourwebsite.com/robots.txt in a browser and confirm the rules show.
  2. Wait about 24 hours.
  3. Run the 30-minute prompt test in how to check if ChatGPT recommends your business.
  4. Look for the source chatgpt.com in Google Analytics: OpenAI adds utm_source=chatgpt.com to links from ChatGPT search.

The restaurant version of this work is in how to get your restaurant recommended by ChatGPT and Google AI.

Note: This guide reflects OpenAI, Google, Perplexity, Anthropic and llmstxt.org documentation as of November 2026. Bots and policies change, so re-check the linked pages.

Frequently Asked Questions