llms.txt and structured data: what actually matters
Two pieces of advice show up in almost every "optimize for AI search" checklist: add an llms.txt file, and add structured data. Both can be worth doing. Neither is the switch some checklists make it sound like. Here's what each one is, what the people who run AI assistants and search engines have actually said about them, and where your time is better spent.
What llms.txt is
llms.txt is a proposal, first published by Jeremy Howard in September 2024 and revised as "v2" in August 2026. It suggests putting a Markdown file at /llms.txt (or at a sub-path such as /docs/llms.txt) that gives a short summary of a site and links to the pages an AI agent is most likely to need, ideally clean Markdown versions of them (llmstxt.org).
The format is simple. An H1 with the site or project name is the only required part. After that come an optional one-paragraph summary in a blockquote, optional notes, and sections headed with H2 that list links with a short description of each. A section called "Optional" holds links an agent can skip when it's short on space (llmstxt.org).
The proposal is clear about what the file is for. It's meant to be read on demand, when an agent needs information while helping someone. It's not an access control. That job stays with robots.txt (llmstxt.org). It says the files are used most heavily for software documentation, where coding agents follow them to find API references.
What's documented about llms.txt, and what isn't
This is where most advice gets ahead of the evidence.
What the proposal itself says: according to its author, thousands of sites publish the file, documentation platforms generate it automatically, and AI labs including OpenAI, Anthropic, and Google's Gemini team publish llms.txt files for their own developer docs (llmstxt.org). You can see one in practice: OpenAI's crawler documentation page points readers to an llms.txt index of its docs (OpenAI), and so do Perplexity's docs (Perplexity).
What the providers say about reading it on your site: that's a different question. Publishing an llms.txt for your own docs isn't the same as saying your search crawler uses other sites' files to decide what to show or cite. On 17 November 2026 we read the crawler documentation from OpenAI, Anthropic, and Perplexity. None of them mentions llms.txt as something their crawlers read or use for search (OpenAI, Anthropic, Perplexity). The controls they document are user agents in robots.txt, plus published IP ranges.
What Google says: Google's guidance for AI Overviews and AI Mode is direct: "You don't need to create new machine readable files, AI text files, or markup to appear in these features" (Google Search Central).
What's commonly believed but unproven: that adding llms.txt makes ChatGPT or other assistants more likely to cite you. We found no provider documentation that supports this.
So should you bother? It's cheap, it can help agents that do read it (coding tools reading developer docs are the clearest case), and an accurate summary of your key pages does no harm. Just don't expect it to change whether you're recommended. If you already have one, the useful check is whether it's accurate.
What structured data is
Structured data is markup, usually JSON-LD using the schema.org vocabulary, that labels what a page is about: an organization, a product, a recipe, an event. Google describes it as "explicit clues about the meaning of a page" and recommends JSON-LD as the easiest format to maintain (Google: Intro to structured data).
What's documented about structured data
For Google Search, quite a lot:
- Google uses structured data to understand page content and to gather information about the people, books, or companies in the markup, and it can enable rich results (Google: Intro to structured data).
- The markup must describe the page it's on. Google says not to mark up information that isn't visible to the user (same page). Its AI-features guidance repeats this: make sure structured data matches the visible text (Google Search Central).
- Fewer, complete, accurate properties beat lots of badly formed ones (Google: Intro to structured data).
Organizationmarkup on your homepage can help Google understand your administrative details and disambiguate your organization. ItssameAsproperty links your profiles on other sites, such as social or review pages (Google: Organization). Schema.org definessameAsas the URL of a page that "unambiguously indicates the item's identity" (schema.org).
And some limits that checklists skip:
- There's no special AI schema. Google says there's "no special schema.org structured data that you need to add" to appear in AI Overviews or AI Mode (Google Search Central).
- FAQ markup isn't what it was. Since August 2023, Google shows FAQ rich results only for "well-known, authoritative government and health websites". Google adds that unused structured data "does not cause problems for Search, but also has no visible effects" (Google Search Central Blog).
- Schema.org is bigger than Google's support. Google says to treat its own docs as definitive for Google Search behavior. Many schema.org types and properties aren't used by Google but may be used by other services (Google: Intro to structured data).
None of the OpenAI, Anthropic, or Perplexity crawler pages we read says how, or whether, they use structured data.
So what actually matters?
In rough priority order, based on what's documented:
- Can the assistant reach the page? OpenAI says sites opted out of OAI-SearchBot won't appear in ChatGPT search answers (OpenAI). Google lists allowing crawling in robots.txt "and by any CDN or hosting infrastructure" as a basic (Google Search Central). No file or markup matters if the page is blocked.
- Is the important content in plain text? Google says to make sure "important content is available in textual form" (Google Search Central).
- Is your structured data the right type, accurate, and matching the page?
OrganizationandWebSiteon a homepage, or aLocalBusinesstype for a shop, beat a pile of generic markup. Validate it with Google's Rich Results Test (Google: Intro to structured data). - Is there an accurate
llms.txt? It's optional, cheap, and helpful for agents that read it. Don't count on it changing recommendations.
None of these guarantees that an assistant will cite you. Google says indexing and serving isn't guaranteed even when a page meets every requirement (Google Search Central), and OpenAI says placement isn't guaranteed either (OpenAI Help).
Check your own pages: CiteLens is a free, no-sign-up audit that reports, with evidence, whether a public page has an llms.txt, which schema.org types its JSON-LD uses, whether robots.txt allows documented AI crawlers, and how its sitemap, title, H1, static text, FAQ coverage, and Open Graph look. It checks readiness, not whether any assistant mentions you.
Disclosure: Born Bots builds CiteLens.
Sources
- Jeremy Howard, "The /llms.txt file, v2" (first published 3 Sep 2024, updated 10 Aug 2026): https://llmstxt.org/ (opened 17 Nov 2026)
- OpenAI, "Overview of OpenAI Crawlers": https://platform.openai.com/docs/bots (opened 17 Nov 2026)
- OpenAI Help Center, "Searching the web with ChatGPT": https://help.openai.com/en/articles/9237897-chatgpt-search (opened 17 Nov 2026)
- Anthropic (Claude Help Center), crawler documentation: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (opened 17 Nov 2026)
- Perplexity, "Perplexity Crawlers": https://docs.perplexity.ai/guides/bots (opened 17 Nov 2026)
- Google Search Central, "AI features and your website" (last updated 10 Dec 2025): https://developers.google.com/search/docs/appearance/ai-features (opened 17 Nov 2026)
- Google Search Central, "Introduction to structured data markup in Google Search": https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data (opened 17 Nov 2026)
- Google Search Central, "Organization structured data": https://developers.google.com/search/docs/appearance/structured-data/organization (opened 17 Nov 2026)
- Google Search Central Blog, "Changes to HowTo and FAQ rich results", 8 Aug 2023: https://developers.google.com/search/blog/2023/08/howto-faq-changes (opened 17 Nov 2026)
- Schema.org, "sameAs": https://schema.org/sameAs (opened 17 Nov 2026)
Last reviewed: 17 Nov 2026.