serkankaya.
Türkçe sürüm
Let’s work together ↗

What Is llms.txt? How to Add It to Your Site in 10 Minutes

What is llms.txt, how is it different from robots.txt, and does anyone read it? File format, Astro and WordPress setup, plus checks and log-based measurement.

What is llms.txt? The short answer: llms.txt is a text file in the site root that presents the site’s summary and most important pages to language models in Markdown. It is not an official standard but a proposal published in 2024. The file does not work as a ranking signal; it is a guide that helps coding assistants and AI agents understand a site quickly.

In this post I cover the llms.txt format, how it differs from robots.txt and sitemap.xml, what the data says about whether it is actually read, and how I add it to this site automatically. If you want the broader picture first, see my post on what GEO is; llms.txt is a small but frequently asked part of the technical side of GEO.

Where did llms.txt come from and why was it proposed?

llms.txt is a proposal published on llmstxt.org on 3 September 2024 by Jeremy Howard, one of the co-founders of Answer.AI.[1] The starting point is simple: web pages are designed for people. Menus, ads, cookie banners and sections loaded with JavaScript make it hard for a language model to reach the actual information on a page.

The second problem is the context window. A context window is the amount of text a language model can process at once. An entire site does not fit into that window, and every unnecessary word costs time and money. llms.txt offers the agent a short front door that says “this is what matters on this site, and the detail is behind these links.”

The proposal was updated to its second version in August 2026.[1] According to the current text, thousands of sites publish an llms.txt file, documentation platforms generate it automatically, and Chrome’s Lighthouse checks for the file in its agentic browsing audits. OpenAI, Anthropic and Google’s Gemini team also publish llms.txt files for their developer documentation. These claims come from the proposal’s own page; they are not an independent measurement.

User question Agent reads llms.txt Opens the relevant link Generates the answer
The intended use of llms.txt: the agent reads the short summary first, then opens only the page it needs.

What format should an llms.txt file follow?

Short answer

What format should an llms.txt file follow?

llms.txt is a plain-text file written in Markdown that follows a fixed order. The only required section is an H1 with the site or project name. It is followed by a short summary in a blockquote, optional detail paragraphs, and link lists under H2 headings in the form “name: note”.

The order in the proposal is:[1]

OrderSectionRequired?What goes in it?
1# Site name (H1)YesThe name of the site, brand or project
2> Summary (blockquote)NoA short summary needed to understand the rest of the file
3Detail paragraphsNoExtra information, notes or caveats, without headings
4## Section + link listNoOne line per link: - [Name](URL): short note
5## OptionalNoSecondary links that can be skipped when context is short

A simple example for a services site:

# Example Agency

> An Istanbul-based digital marketing agency providing technical SEO and Google Ads management.

We work in Turkish and English. Prices are not listed on the pages; quotes are given via the contact form.

## Services

- [Technical SEO](https://example.com/technical-seo/): Site audits, crawling and indexing issues
- [Google Ads management](https://example.com/google-ads/): Campaign setup and conversion tracking

## Guides

- [Core Web Vitals guide](https://example.com/blog/core-web-vitals/): Measuring page speed and steps to improve it

## Optional

- [Privacy policy](https://example.com/privacy/)

The format is Markdown because the file should be easy to read for both people and language models. Since the structure is still fixed, it can also be processed with classic software tools, such as a parser or a regular expression.

Subdirectories and Markdown versions

The second version of the proposal adds two details.[1] First, the file can sit in a subdirectory, not only in the root. For example, /docs/llms.txt covers only the pages under /docs/, and when more than one file applies, the agent uses the most specific one. Second, a clean Markdown version of pages an agent may need can be served at the same URL with .md appended. On top of that, some documentation platforms publish an llms-full.txt that merges all content into one file; that file is not defined in the proposal itself, it is a practice that has become common.

To point agents to these versions, the proposal recommends standard link relations. rel="alternate" type="text/markdown" points to the Markdown version of a page, and rel="describedby" points to the llms.txt file that covers it. These relations can be given with a <link> element in HTML or with an HTTP Link header; the header form relies on the web linking standard defined in RFC 8288.[4]

What is the difference between llms.txt, robots.txt and sitemap.xml?

llms.txt sets no access rules; robots.txt governs where bots may go, and sitemap.xml lists every indexable page. The three files answer different questions, and none replaces another. Adding a section you block in robots.txt to llms.txt sends the agent a contradictory signal.

robots.txtsitemap.xmlllms.txt
Question it answersWhere can the bot go?Which pages exist?What should be read first?
Target readerCrawler botsSearch enginesLanguage models and agents
FormatPlain-text rulesXMLMarkdown
ScopeWhole siteAll indexable pagesSelected key pages
StatusEstablished standardEstablished standardCommunity proposal

The proposal frames the relationship the same way: sitemap.xml lists all pages and is usually too large to fit in a context window, while llms.txt offers a curated summary.[1] In other words, llms.txt is not a shortened copy of the sitemap; it is an editorial selection.

On the robots.txt side, what really matters is not accidentally blocking the search bots of the platforms you want to appear in. I covered that distinction in detail, using the OpenAI and Perplexity bots, in my post on what GEO is.

Does llms.txt actually work?

There is no evidence that llms.txt has a direct effect on search visibility, and Google says so explicitly. According to Google’s documentation on AI features, you do not need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode.[2] It is enough for the page to be indexed and eligible to be shown with a snippet.

Usage data also lowers expectations. In a study Ahrefs published in June 2026, 97% of roughly 38,000 domains with a valid llms.txt file received no requests for that file at all during May 2026.[3] For the files that were read, 96% of requests came from bots.

Domains with a valid llms.txt file
~38,000
Share of files that received no requests in May 2026
97%
Share of requests from bots on files that were read
96%
Source: Ahrefs, llms.txt study (June 2026)

So why do I still add it? I have three reasons:

  1. The cost is very low. Once the file is set up and automated, it updates itself as content is added.
  2. There is a real use case. According to the proposal, the file is used most in software documentation, where coding assistants follow it to reach API references.[1] On a services site, an agent can also answer “what does this person do?” from a single file.
  3. It makes you sharper. Summarizing your site in one paragraph and choosing its ten most important pages is a useful exercise for content strategy too.

How do you create an llms.txt file?

Short answer

How do you create an llms.txt file?

To create llms.txt, write the site name, a short summary and links to the most important pages into a text file in Markdown, then upload it to the site root. You can write the file by hand, generate it with a CMS plugin, or produce it automatically during the build.

  1. Write the site name and a one-paragraph summary.
  2. Pick the 10-20 most important pages and add a short note to each.
  3. Group the pages under H2 headings.
  4. Upload the file to the site root as llms.txt.
  5. Check that the URL returns 200 and is served as text/plain.

If you would rather not write it by hand, the free llms.txt generator on the Turkish version of this site turns a filled-in form into the structure from the proposal and lets you download it. The lasting solution, though, is to generate the file from the site’s own content.

  1. Write the summary

    Say who you are, what you do, and whom and where you serve in two sentences. Write information, not a marketing slogan.

  2. Pick the pages

    Service and product pages, your most comprehensive guides, the about and contact pages. Leave out tag archives, cart and account pages.

  3. Add the notes

    Give every link a short description of what the page answers. The agent decides which link to open based on this note.

  4. Publish and check

    Put the file in the root, open it in a browser and check the HTTP response. Then give an agent only this file and ask it questions about your site.

Automatic generation with Astro

This site is built with Astro and deployed on Cloudflare Pages. Instead of maintaining llms.txt by hand, I generate it at build time with an Astro endpoint. In Astro, an endpoint is a .ts file inside src/pages that returns a response of any type you want instead of HTML.[5] A simplified version looks like this:

// src/pages/llms.txt.ts
import { getCollection } from 'astro:content';

export async function GET() {
  const posts = await getCollection('yazilar', ({ data }) => !data.draft);
  const lines = [
    '# Site name',
    '',
    '> A two-sentence summary of the site.',
    '',
    '## Posts',
    ...posts.map((p) => `- [${p.data.title}](https://yourdomain.com/yazilar/${p.id}/): ${p.data.description}`),
  ];
  return new Response(lines.join('\n'), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

On an Astro site with static output, this endpoint runs once at build time and the result is published as a plain file. When I add a new post, the file updates itself on the next build. In this site’s /llms.txt file, service pages, sector pages and the English versions are listed with the same logic alongside the posts.

WordPress and other platforms

On WordPress, the easiest route is an SEO plugin. Among the integrations that generate llms.txt, the proposal site lists the Yoast SEO and AIOSEO plugins, Wix, and on the documentation side Mintlify and GitBook.[1] If you do not use a plugin, you can write the file by hand and upload it to the root via FTP or the file manager in your hosting panel. On hosted site builders, the ability to upload files to the root may be limited; in that case you need to check the platform’s own documentation.

Server and CDN settings

Having the file live is not enough; it also has to be served correctly. I pay attention to three points:

  • Content type: The Content-Type header of the response should be text/plain; charset=utf-8. If Turkish or other non-ASCII characters look broken, the cause is usually a missing charset.
  • Bot protection: Bot rules in CDN and WAF (Web Application Firewall) layers such as Cloudflare can close the file off to agents. Leave the file on an unblocked path.
  • Link header: On static sites, a file such as _headers can add a Link: </llms.txt>; rel="describedby" header to pages. That way you can show agents where the file is without changing the pages.[1]

Which pages should go into llms.txt?

llms.txt should contain not the whole site but the most important pages an agent needs to understand the site correctly. In practice that means service or product categories, your most comprehensive guides, and the about and contact pages. The point of the file is to stay short enough to fit in a context window; the detail should sit behind the links.

What to leave out matters just as much:

  • Personal data: The file is public text. Do not include staff phone numbers, email lists or customer information. Under KVKK (Turkey’s data protection law) and GDPR it is also an unnecessary risk.
  • Private or admin pages: Anyone can see every URL you put in llms.txt. Do not list admin panels, staging environments or paths you block in robots.txt.
  • Duplicate pages: Tag archives, filter results, pagination URLs and checkout steps give the agent no new information.
  • Unverified claims: The summary paragraph is not a slogan space. Unprovable phrases like “the industry leader” can lead the model to describe you incorrectly.

On a Turkish site, writing the summary in Turkish is the natural choice. If your site also has English pages, listing them under a separate H2 heading makes it easier for the agent to reach the right language version. On this site, too, the English pages sit separately under an “English” heading.

How do you check whether llms.txt is being read?

The most reliable way to tell whether llms.txt is read is to inspect the requests to /llms.txt in your server or CDN logs, along with their user agents. Analytics tools do not show these requests, because the file does not run JavaScript. The 97% figure in the Ahrefs study is based on exactly this kind of request data.[3]

In my own setup I work in this order: first I check that the file is served correctly, then, following the advice in the proposal, I give an agent only the llms.txt file and ask it questions about the site.[1] If the agent goes to the right page and gives the right answer, the summary and notes are doing their job. Last, I look at the logs to see who requested the file.

llms.txt pre-launch checklist0/8

When reading logs, remember that user agent strings can be spoofed. For important decisions, it is safer to verify bots with your CDN’s own tools or with the IP ranges the platforms publish.

Where should you start?

llms.txt is a file that summarizes your site for language models and agents; it takes little time to set up, but its impact is limited. Google does not treat it as a requirement, and current data shows that most of these files are never read. Even so, when it is generated automatically its maintenance cost is close to zero, so adding it is a sensible choice.

My suggestion is to go in this order: first make sure your robots.txt and CDN settings do not block the bots you want to be seen by. Then prepare your first llms.txt with a two-sentence summary and your most important pages, and connect it to your CMS or build process if you can. Finally, watch the requests in your logs. Wiring this kind of automation into a site’s infrastructure is software development work; I covered the rest of AI visibility in my post on what GEO is.

Key terms

robots.txt
robots.txt is a plain-text rules file in the site root that tells crawler bots which URLs they may or may not access. It is not a security measure; whether a bot follows the rules depends on the bot itself.
sitemap.xml
sitemap.xml is a file that lists the indexable pages of a site for search engines in XML format. It makes pages easier to discover but does not say which pages matter most or how they should be interpreted.
Markdown
Markdown is a lightweight markup language for writing headings, lists and links with simple characters such as # and -. It was chosen as the format for llms.txt files because it is easy for both people and language models to read.
llms-full.txt
llms-full.txt is a file some documentation platforms publish alongside llms.txt that combines all of a site's content into a single Markdown file. It is not defined in the llms.txt proposal itself; it is a practice that has become common.

Frequently asked questions

What is llms.txt?

llms.txt is a text file at the root of a website that gives language models and AI agents a short summary of the site and its most important pages in Markdown. It is a proposal Jeremy Howard published in 2024, not an official web standard.

What is the difference between llms.txt and robots.txt?

robots.txt is an access rules file that tells bots which parts of a site they may crawl. llms.txt sets no access rules; it offers a summary and a curated list of links so an agent can understand the site quickly. Neither replaces the other, and they can be used together.

Does llms.txt affect SEO?

Google states in its official documentation that you do not need to create new machine-readable files or AI text files to appear in AI Overviews and AI Mode. So it is more accurate to treat llms.txt as a low-cost convenience for coding assistants and agents than as a ranking signal.

Where do you upload an llms.txt file?

The file usually goes in the site root, at yourdomain.com/llms.txt. The current version of the proposal says it can also sit in a subdirectory and then covers only the pages under that path; for example, /docs/llms.txt describes only what is under /docs/.

How do you create llms.txt in WordPress?

The easiest route is an SEO plugin that supports llms.txt generation; the proposal site lists Yoast SEO and AIOSEO among these integrations. If you prefer not to use a plugin, you can write the file by hand and upload it to the root via FTP or your hosting file manager.

Sources

  1. The /llms.txt file (Jeremy Howard, llmstxt.org)
  2. Google Search Central: AI features and your website
  3. Ahrefs: We Analyzed 137K Sites, 97% of llms.txt Files Never Get Read
  4. RFC 8288: Web Linking
  5. Astro Docs: Endpoints