llms.txt, robots.txt and ChatGPT: What Your UAE Website Actually Needs (2026)
To appear in ChatGPT search, your website must allow OpenAI's OAI-SearchBot in robots.txt and in any host or CDN security layer. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that any public website can appear, with placement not guaranteed (OpenAI crawler documentation). llms.txt is optional. It is a community proposal, Google says its search ignores the file, and we found no statement in OpenAI's, Anthropic's or Perplexity's crawler documentation that their crawlers read it. Fix robots.txt first; treat llms.txt as a 15-minute extra at most.
Which AI Bots Should Your Website Allow?
| Company | Bot | What the company says it is for | If you block it |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfacing websites in ChatGPT's search features | Not shown in ChatGPT search answers |
| OpenAI | ChatGPT-User | User-initiated actions, not automatic crawling | Does not decide search eligibility; that is controlled by OAI-SearchBot |
| OpenAI | GPTBot | Making OpenAI's generative AI foundation models more useful and safe | Opts out of training; independent of search |
| Perplexity | PerplexityBot | Surfacing and linking websites in Perplexity search results | Perplexity recommends allowing it to appear in search results |
| Perplexity | Perplexity-User | Supporting user actions; not used for crawling or training | Separate from PerplexityBot |
| Anthropic | Claude-SearchBot | Indexing content to improve search result quality | May reduce visibility and accuracy in user search results |
| Anthropic | Claude-User | User-directed requests | May reduce visibility in user-directed web search |
| Anthropic | ClaudeBot | Collecting web content that may contribute to training | Opts out of training |
| Googlebot | Google Search, including AI Overviews and AI Mode | Pages cannot be indexed, so they are not eligible for AI features |
Sources: OpenAI, Perplexity, Anthropic, Google. OpenAI says each of its settings is independent of the others, so you can allow search and still block training.
What Does a Working robots.txt Look Like?
This version allows everything. Replace the sitemap line with your own.
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xml
To stay in search but opt out of training, add these two blocks. OpenAI and Anthropic both say their bots follow robots.txt.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Anthropic notes that rules must be set for each subdomain separately, and that blocking IP addresses may not reliably opt a site out.
Is Your Host or CDN Blocking the Bots?
robots.txt is only half of access. Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as its first AI-search best practice (Google Search Central). Some website builders and security services block unknown bots by default. Ask your web host or CDN provider whether they do, and ask them to allow OpenAI's search bot. OpenAI publishes the IP addresses its bots use.
After you change anything, wait. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
What Is llms.txt, and Should You Publish One?
llms.txt is a proposal by Jeremy Howard, first published on 3 September 2024. Its own page calls it "A proposal to standardise on using an /llms.txt file," open for community input, not a formal standard (llmstxt.org). It is a Markdown file with:
- one H1 with the site name (the only required part)
- an optional short summary in a blockquote
- H2 sections with lists of links, each as
[name](url)plus an optional note
Does it work? Google says no: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search," and llms.txt files "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them" (Google Search Central). For OpenAI, Anthropic and Perplexity, we found crawler documentation covering robots.txt but no statement that their crawlers read llms.txt. The proposal says the file is meant for agents to read, but that is the proposal's intent, not a platform commitment.
Our view: it is harmless, so publish one if it takes you 15 minutes. EvolvXAI publishes one for this blog. Do not pay a service for it, and do not let it replace the work that Google and OpenAI do document: crawler access, a complete Business Profile, and key facts as text.
Example for a restaurant:
# Example Restaurant
> Family restaurant serving North Indian food in Al Barsha, Dubai. Halal. Parking available.
## Key pages
- [Menu](https://yourwebsite.com/menu): Dishes and prices
- [Opening hours and location](https://yourwebsite.com/contact): Address, hours, parking
- [Book a table](https://yourwebsite.com/book): Reservations
How Do You Test That It Worked?
- Open
yourwebsite.com/robots.txtin a browser and confirm the rules show. - Wait about 24 hours.
- Run the 30-minute prompt test in how to check if ChatGPT recommends your business.
- Look for the source chatgpt.com in Google Analytics: OpenAI adds
utm_source=chatgpt.comto links from ChatGPT search.
The restaurant version of this work is in how to get your restaurant recommended by ChatGPT and Google AI.
Note: This guide reflects OpenAI, Google, Perplexity, Anthropic and llmstxt.org documentation as of November 2026. Bots and policies change, so re-check the linked pages.