Make your site readable by AI agents

AI agents visit websites on users' behalf to answer questions, integrate APIs, and complete tasks. Together with the ora.ai research lab, we observed 1,033 agent runs across 25 sites and ran 190 controlled fetch probes. The trace study shows where agents went and which fetched pages grounded their answers. The controlled experiment tests which page configurations allowed two fetch clients to retrieve a planted answer.

In that controlled experiment, two kinds of change prevented answer retrieval. One hid the answer behind JavaScript from a fetch-only client. The other returned 403 to both clients. Agents must find and read a site's content before they can act on it.

These results describe this sample rather than the web as a whole. The trace results are observational and run-weighted. Run counts vary by site, model, and harness. Six sites account for 691 of 1,033 runs, or 66.9%. See the data. The controlled probes test whether an answer is present in the returned content, not whether an agent understands it or completes a task.

This guide reflects data collected from June through August 2026. Ora continues to expand the datasets with more sites, models, harnesses, and probes. Future guide versions may update the figures and recommendations as the sample grows.

Every recommendation depends on the agent reaching the page and finding the answer in the returned HTML.

Copy link to headingBe reachable and citable

To reliably cite current first-party information, agents need a discoverable, fetchable page. They may receive or recall a URL, follow a link, or find a page through a crawl-built search index. Access controls can stop an agent from fetching a page while answering a user.

Publish each answer on one canonical, current first-party docs page. In these runs, docs were the most common grounding source. Of the 844 grounded answers, 394, or 47%, traced to a docs page.

Public sites do not need a robots.txt file to permit crawling. If your site already uses one for deliberate exclusions, audit it separately rather than adding agent-specific rules by default.

Then fetch a page with an agent user-agent string to inspect the response:

curl -iL -A 'Claude-User/1.0' https://example.com/your-page

The request should end with a 200 response. Intermediate 301 or 308 redirects are acceptable. The returned HTML should contain the answer. A 403, empty app shell, or JavaScript-only answer prevents fetch-only agents from reading the page.

This curl request cannot confirm whether a real agent is blocked. Bot blockers weigh where a request comes from more than its user-agent string, so your laptop may pass where the real agent is blocked, or the reverse. Server logs are the best operational check. OpenAI and Anthropic publish crawler IP ranges. Find requests from those ranges and confirm that they reach the intended page and receive the expected response. A 200 confirms access, but not that the returned content contains the answer or that the agent interpreted it correctly.

Agents can use llms.txt only after discovering and reaching the site.

Copy link to headingPut the answer where agents read

After reaching a page, an agent must be able to extract the answer from the returned content. In the studies, agents used the same homepage and docs pages as human visitors. Agent-specific resources such as llms.txt, Markdown mirrors, and JSON-LD can help agents discover or consume the same underlying information. They do not replace accurate, fetchable source pages.

If your site is client-rendered, making answers available in the initial HTML may require server-side rendering (SSR), prerendering, or a hybrid approach so agents that do not run JavaScript can read them. In the controlled probes, adding llms.txt, JSON-LD, sitemaps, Markdown mirrors, and similar features did not change whether the clients could retrieve an answer already present in server-rendered HTML.

Copy link to headingAgents read the same pages people do

Where agents go on your site, and what lets them pull the answer once there:

Copy link to headingLink the files that describe your site

[STUDIES] Depending on the file type, 86-97% of attributed fetches came through links rather than guessed paths. See the data.

Link each resource from content agents already visited, such as the footer, page metadata like <link rel="alternate"> in the head, or a resources section.

When shipping an llms.txt, make it a lean, described index:

# Acme

> Acme ships a payments API for developers.

## Docs

- [Quickstart](https://example.com/docs/quickstart): first charge in one request
- [Authentication](https://example.com/docs/auth): API keys, scopes, rotation

## API

- [OpenAPI](https://example.com/openapi.json): full request and response contract

Add JSON-LD where it accurately represents the page, including the homepage when it contains organization or product facts. Keep critical information in visible text as well. This is a structured-data recommendation rather than a measured outcome from these studies.

Copy link to headingAgents frequently requested Markdown in the study harnesses

[STUDIES] Across the study harnesses, 2,259 of 3,470 web fetches, or 65.1%, requested Markdown. When a fetch specified any format, it selected Markdown in 2,259 of 2,361 cases, or 95.7%. In the controlled experiment, every non-blocking configuration exposed the planted answer to the applicable fetch client. See the data. Make the HTML work before adding a Markdown mirror.

<!-- docs/quickstart.html -->
<!-- Advertise the mirror so agents can discover it. -->
<link rel="alternate" type="text/markdown" href="/docs/quickstart.md">

You can also provide the same relation in an HTTP Link header, allowing a CDN to advertise the mirror without changing page markup.

Copy link to headingReturn accurate HTTP status responses

Return 404 for a missing path or a valid redirect to a replacement. Do not return 200 for a missing page, even if the body suggests alternatives. A soft 404 removes the reliable HTTP signal that distinguishes missing content from a live page.

When you throttle, return 429 Too Many Requests with a Retry-After header instead of a silent challenge page. This gives clients an explicit, standards-based back-off signal.

If you offer Markdown through content negotiation, keep the response metadata accurate. A request with Accept: text/markdown should receive Markdown with Content-Type: text/markdown; charset=utf-8. Include Vary: Accept so caches keep the HTML and Markdown responses separate. If Markdown is unavailable, return HTML with its correct content type.

Copy link to headingEnable agents to act

If agents need to operate your product, expose the required actions through an authenticated API, MCP server, SDK, or CLI, and document the complete path from authentication to the first successful action.

Prioritize authentication an agent can complete and a public API with a machine-readable OpenAPI contract. For authentication, provide self-serve API keys with documented scopes or discoverable OAuth 2.0 metadata under /.well-known. [STUDIES] On the seven sites that shipped an openapi.json, agents fetched it in 112 of 541 runs, or 21%. See the data.

The studies measured OpenAPI retrieval, not successful authentication or task completion. Follow the relevant protocol specifications and test the complete workflow with the agent clients you intend to support. Use a stable endpoint such as /mcp for a Streamable HTTP MCP server, but do not treat that path as a protocol requirement.

Copy link to headingRun additional agent-readiness checks

Agent readiness includes practices beyond just those that we have covered here. Use is-agentic.com to check domains against a wider set of tests. The Ora research lab runs and scores every scan. Expect the checks and their weights to shift as new agent behavior gets measured.

Paste a domain into the site to see your evaluation, or pull a stored report from the API with a GET request:

curl -s 'https://is-agentic.com/api/v1/report?url=vercel.com'

Abridged, a response looks like this:

{
  "target": "https://vercel.com",
  "report_url": "https://is-agentic.com/scan/vercel.com",
  "score": 89,
  "score_label": "Strong technical baseline",
  "eligible_checks": 33,
  "score_breakdown": {
    "essential": { "earned": 65.9, "available": 80, "passing": 8, "total": 11 },
    "recommended": { "earned": 18.2, "available": 20, "passing": 18, "total": 22 },
    "bonus": { "points": 5, "positive_signals": 39 }
  },
  "issues": [
    { "id": "agent-friendly-404", "name": "Agent-friendly 404s", "tier": "essential", "result": "failed" }
  ]
}

Scores move whenever a domain is rescanned, so treat any single number as a snapshot. Each issue shows the evidence behind it and suggests a fix. You can pull the same report from npx is-agentic <domain> or a read-only MCP server at is-agentic.com/mcp.

Copy link to headingReferences

• What agents actually reach: an earlier field report from the same research program. The published dataset below is the source of the figures in this guide.
• ora research: the ongoing research program behind the protocol.
• Derived agent-run summaries and controlled fetch probes: the data behind every [STUDIES] number in this guide.

Copy link to headingChangelog

• v1.0 · August 2026 · first public version.

Copy link to headingProvenance

Copy link to headingAnswer provenance

Backs: "Of runs with an answer-provenance result, 82.5% traced to a fetched page. Among grounded answers, 47% traced to a docs page."

Of the 1,023 runs with an answer-provenance result, where did the agent's final answer come from? Ten additional runs were marked not applicable.

Of the 844 grounded answers, the page types the wording traced to (top 3):

Docs lead every other type. The full 14-way split is in traces.csv.

Reproduce over traces.csv. Among rows where answer_grounded is set, use answer_grounded == 1 to identify grounded answers. Use answer_ground_kind for the page type.

Copy link to headingSite weighting

Backs: "Six sites account for 691 of 1,033 runs, or 66.9%."

The six sites with the most runs were:

Reproduce over traces.csv: group rows by domain, sort run counts in descending order, and sum the first six.

Copy link to headingSitemap reach

Backs: "Fetched directly in 4% of runs."

Reproduce over traces.csv: sitemap_reached == 1.

Copy link to headingHomepage reach and first hop

Backs: "Reached in 69% of runs, and it was the agent's first step in 92% of them. In 59% of the runs that reached it, the next hop was a docs page."

The homepage funnel across 1,033 runs:

Reproduce over traces.csv: homepage_reached, homepage_first_turn, and next_hop_docs, each rated against homepage_reached == 1.

Copy link to headingDocs reach and depth

Backs: "Reached in 83% of runs. Among docs-reaching runs, agents fetched an average of 3.4 docs pages."

Reproduce over traces.csv. Select rows where docs_reached == 1, then calculate the mean of docs_pages_fetched over those runs.

Copy link to headingFetchability: answer retrieval

Backs: "In the controlled probes, a fetch-only client could not retrieve an answer available only through JavaScript. Neither client could retrieve the answer when the server returned 403."

One docs site was served under 19 configurations and probed across 5 topics by two clients: a plain non-JavaScript fetch and a JavaScript-executing one (190 probes). Only three configurations prevented at least one client from retrieving the answer, and they represent two kinds of change:

A JavaScript-hidden answer is unavailable to the fetch-only client. Blocking prevents both clients from retrieving the answer.

Reproduce over fetchability.csv: success grouped by variant and client.

Copy link to headingCode lift

Backs: "When an agent's answer included code, 61% of the time it contained lines taken verbatim from a page it fetched on the site."

Among the 119 runs whose answer contained code:

Reproduce over traces.csv: among rows where answer_code_matched_fetch is set, share == 1.

Copy link to headingDiscovery-file linkage

Backs: "Depending on the file type, 86-97% of attributed fetches came through links rather than guessed paths."

For each discovery file, the share of fetches that arrived through a link rather than a guessed path:

These values count fetches, not unique runs. A run can fetch the same file more than once, so the 122 openapi.json fetches above correspond to 112 runs in the openapi reach section.

Reproduce over traces.csv: per file, {file}_linked / {file}_fetches.

Copy link to headingllms.txt behavior

Backs: "Agents reached llms.txt through links in 356 of 416 attributed fetches, or 86%. The steering and answer-grounding measures use different denominators."

How agents reached llms.txt (416 fetches):

Of the 329 runs that reached llms.txt, 325 had a scorable steering result. The answer-grounding measure is available across all 329 runs:

Reproduce over traces.csv. Calculate via-link as llms_txt_linked / llms_txt_fetches. Calculate steering from llms_followed_listing where llms_txt_reached == 1 and llms_followed_listing is set. Calculate answer-from-llms as answer_ground_kind == 'llms_txt' over llms_txt_reached == 1.

Copy link to heading.well-known reach

Backs: "Agents reached a .well-known path in 234 of 1,033 runs, or 22.7%."

Reproduce over traces.csv: well_known_reached == 1.

Copy link to headingopenapi reach

Backs: "On the seven sites that shipped an openapi.json, agents fetched it in 112 of 541 runs, or 21%."

Restricted to the 7 sites that ship openapi.json (541 runs):

Reproduce over traces.csv: openapi_reached == 1 over site_ships_openapi == 1.

Copy link to headingMarkdown requests

Backs: "Agents requested Markdown in 2,259 of 3,470 web fetches, or 65.1%. When they specified a format, they chose Markdown in 2,259 of 2,361 fetches, or 95.7%."

Reproduce over traces.csv. Calculate markdown_calls / web_fetch_calls and markdown_calls / format_set_calls.