One Token, Four Gatekeepers: What the New AI Opt-Out Standard Means for Indexing, llms.txt, and AI Citations
OpenAI, Google, Microsoft, and Anthropic have said they will honor a shared opt-out signal that publishers can set once and have applied across all of them, as reported by The Verge (following Search Engine Roundtable). Instead of configuring a separate directive for every vendor's crawler, a publisher applies a single "policy token" expressing their AI training preference. Details of the syntax are still landing, but the shape of the thing is clear — and it's the first time the four largest commercial AI operators have agreed to read the same permission surface.

Figure: Editorial context image from the source page (preview only; rights remain with the publisher) · Credit / further reading: www.theverge.com
That sounds like a small plumbing change. It isn't. It forces every publisher to answer a question most have been avoiding: opt out of what, exactly?
The problem it solves is a config-drift problem
robots.txt has quietly become a registry of vendor-specific user agents. GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, ClaudeBot, Claude-User, CCBot, Bytespider, PerplexityBot, Applebot-Extended — each vendor invents a token, revises it, and renames it. Meanwhile the file itself is edited twice a year by whoever remembers, with no owner, no review, and no test.
The result is three predictable failure modes:
- Coverage gaps. You block six of nine relevant agents and believe you're covered.
- False safety. You block bulk training but forget live retrieval, or vice versa.
- No audit trail. Nobody can reconstruct why a given directive exists or who approved it.
A shared token collapses a many-to-many negotiation into one declaration. Operationally, that makes crawler policy a versioned config item — which means it should be owned, reviewed, tested, and logged like one.
Three pipelines, and the token only speaks to one
The expensive mistake here is treating "AI" as a single thing. There are at least three distinct pipelines, and they have different economics:
- Classic search indexing (Googlebot, Bingbot). Drives rankings. Not what this standard is about.
- AI training — bulk crawl, dataset assembly, no attribution, payoff measured in years.
- AI answer retrieval / grounding — a live fetch to answer a specific query, usually with a citation. This is where AI-surface traffic actually comes from.
A token saying "no training" is not a token saying "no citation." Historically, Google-Extended covered Gemini training and grounding but explicitly did not affect Search — meaning a site that opted out could still surface in AI Overviews via normal indexing. Vendors draw that line in different places, and the new standard's entire value depends on whether it makes the boundary explicit and uniform.
Plan on this asymmetry: training opt-out is cheap; retrieval opt-out is expensive. Blocking retrieval removes you from the answer layer, and the answer layer is increasingly the front page. Decide per-pipeline and write the decision down. "We opted out of AI" is not a strategy; it's a shrug.
Where the token and llms.txt diverge
llms.txt is frequently described as "robots.txt for AI." That's a category error, and this announcement makes it obvious.
- A policy token is a permission signal. Near-binary, machine-enforced (in theory), about what an operator may do. It sits in the same layer as robots.txt.
- llms.txt is a curation signal. A markdown index of your best pages, with short descriptions. No major AI vendor has confirmed they read it, and nothing in the new opt-out standard changes that.
They're complementary and neither substitutes for the other. A token without llms.txt says "here are my terms, work out the content yourself." llms.txt without a token says "here's a helpful map, with no enforceable conditions attached."
The practical read: keep publishing llms.txt — it's cheap, has no real downside, and helps human and agentic consumers orient — but stop treating it as a control. The token is the control. Neither one earns you a citation.
Structured content is the actual lever
If retrieval isn't going away, the operative question becomes: what makes a page quotable?
- Answer-shaped blocks. A 40–60 word direct answer under an H2 that restates the question in the user's words.
- Schema.org markup that matches reality.
Article/NewsArticle,Organization,Personfor authors,datePublished/dateModified,FAQPagewhere the content genuinely is Q&A. Markup that contradicts the visible page is worse than none. - Entity consistency. Same organization name, same byline, same canonical URL everywhere. Retrievers resolve entities; ambiguity loses.
- Facts in the passage itself. Numbers, dates, and named sources travel well through extraction. Vibe prose doesn't.
- Clean HTML. If the answer lives in a client-rendered accordion or inside an image, it isn't in the retrieval corpus at all.

Figure: HEXOPS original illustration inspired by the topic (not a copy of the source article imagery) · Credit / further reading: www.theverge.com
Structured metadata doesn't earn citations directly. It changes what a system can verify about you — and when a model is choosing three sources from twenty candidates, verified beats unverified.
A 90-minute checklist
- Inventory every bot directive: robots.txt, meta robots, WAF and bot-management rules, CDN configs. Most organizations find at least two layers that contradict each other.
- State a position per pipeline — training (allow/deny), retrieval and citation (usually allow), classic search (allow) — and get sign-off, because you will be asked again.
- Adopt the shared token once syntax is settled, but keep vendor-specific entries as a fallback for operators that lag.
- Add it to change management. Crawler policy is infrastructure config. A silent edit is an incident.
- Instrument the outcome. Log AI referrers, track citation appearances, and watch for a traffic step-down if an operator stops fetching you.
The open questions worth watching
- Verification. Will there be a public checker, or will you discover you were ignored by watching traffic? Without verification, "we will adopt" is a promise, not a control.
- Granularity. Does the token support "no training, yes retrieval"? If it's all-or-nothing, most publishers will decline to use it.
- Lag. Vendors ship parsers on their own schedule; expect months of partial coverage and a long tail of smaller operators that never adopt.
- Regulatory overlap. With EU text-and-data-mining reservations and licensing markets in play, a private standard may become the de facto compliance artifact — which raises the stakes on getting the syntax right the first time.
Bottom line
The shared token is a real improvement in the plumbing: one declaration instead of nine, an auditable config instead of folklore. It doesn't resolve the strategic question. Training opt-out is cheap and largely symbolic; retrieval opt-out is costly and, for most publishers, self-harming. So use the token to state your terms unambiguously, use llms.txt to curate, and use structured, extractable content to earn the citations you actually want.
Sources
- The Verge — The biggest names in AI are backing a new way to opt out of scraping: https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out
- Search Engine Roundtable (original reporting credited by The Verge): https://www.seroundtable.com
- llms.txt proposal and specification: https://llmstxt.org
- Google Search Central — Google crawlers and user-triggered fetchers (Google-Extended scope): https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Schema.org vocabulary reference: https://schema.org



