How we made wayf.ai readable by machines
A few weeks after we welcomed every crawler in robots.txt, we fetched the live file: Cloudflare was blocking the training bots at the edge. The audit that followed — discoverable, understandable, trustworthy, measurable — with the receipts, the failures, and the checks you can run on your own site.
A few weeks after we added a wildcard rule welcoming crawlers to wayf.ai, we fetched the live robots.txt.
The repository allowed every crawler. Cloudflare had inserted a managed block ahead of our file that denied training crawlers including ClaudeBot, GPTBot, and Google-Extended.
The public response was more restrictive than the source file. Retrieval bots including ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, and PerplexityBot remained allowed.
That finding changed how we think about machine readability. A crawler has to find a page, retrieve its substance, understand its structure and entities, and encounter claims that agree across the page and its machine-readable forms. Each link in that chain needs a production check.
We ran that audit on wayf.ai. Here is what changed, how we check it, and where the evidence stops.
1. Discoverable: find and retrieve the page
The first test is a fetch:
curl -L https://wayf.ai/some-page/ | grep "a fact from the page"A useful fact should be present in the response body. We prefer to ship the substance as HTML.
wayf.ai is built with Astro and produces static pages. Article bodies, headings, links, tables, and structured data are in the built HTML. Interactive code does not assemble the article a fetcher came to retrieve.
We check rendered output from a journal article, glossary entry, and service page.
One page, one durable address
Our Astro site URL drives canonical tags, Open Graph URLs, and the sitemap. When we moved the primary domain from wayfdigital.com to wayf.ai, we changed that value and preserved paths through permanent redirects.
Cloudflare Pages treated /old-path and /old-path/ as separate matches in our rules. A rename that covered one could leave the other serving an old page or returning a 404. We now test both forms.
We also found that a 120-rule redirects file stopped working near the end. Cloudflare documented a much higher limit, but production dropped rules after roughly the hundredth entry. Fifteen old /work/ URLs with trailing slashes returned 404s while their slash-less equivalents still redirected.
The repair replaced 70 static rules with two dynamic ones. After a deployment that changes redirects, we curl a rule near the bottom of the file. The deployed response decides whether the migration worked.
Give crawlers explicit maps
Astro generates the sitemap. One RSS feed covers the blog and journal and advertises itself in the page head. IndexNow gives participating search engines a direct notification when a meaningful URL changes. Submission sits in our release-review prompt, which sends the request when the change closes.
We also maintain /llms.txt: a curated map with a description beside each selected link. We update it when substantial content ships.
We do not treat llms.txt as a ranking lever. Google Search says it does not use these files, including for its generative features. Support elsewhere remains unclear. We keep ours because it is cheap to maintain and doubles as a compact description of the site.
Check crawler policy at the edge
Our repository’s robots.txt documents the intended policy and applies Allow: / to the wildcard group. Its comments name search, retrieval, and training crawlers separately.
Cloudflare AI Crawl Control then inserted explicit disallow rules for training crawlers into the live response. Those named groups overrode our wildcard allow. The same response left retrieval bots without a specific disallow. Reading public/robots.txt showed our broad intent. Fetching https://wayf.ai/robots.txt showed the policy each crawler received.
That check now belongs beside the source review:
curl -s https://wayf.ai/robots.txtThe same check applies to headers, redirects, canonicals, and rendered HTML. Infrastructure can change the public artifact after the build.
2. Understandable: identify the page and its entities
Semantic structure preserves the relationships in the copy.
Our article and glossary templates use heading hierarchies, lists for sequences, and real tables for comparisons. Strip the styling and read the outline: it should still explain the page.
This matters on glossary comparison pages. A <table> preserves the relationship between headers and values in the document itself.
A shared entity graph
The site layout emits an Organization node and a WebSite node in JSON-LD. The organization has a stable identifier:
https://wayf.ai/#organizationPage-level schemas reference that identifier. Blog and journal entries use BlogPosting; the glossary uses DefinedTerm and DefinedTermSet; service pages use Service and BreadcrumbList; role pages use JobPosting.
The company entity has one node. Page schemas point to it instead of restating it.
We validate rendered pages because the right object in source code can still produce broken output.
One FAQ source for people and machines
Our glossary, blog, and journal templates take FAQs from a frontmatter array. The same questions and answers render the visible accordion and the FAQPage JSON-LD. There is no second block of schema copy for an editor to remember.
That removes one source of drift. The shared FAQ component serializes its JSON-LD through safeJsonLd(), which escapes < so content cannot terminate the script element early.
Content fields have jobs
Our content collections are schema-validated. A glossary entry can carry a shortDef, FAQs, sources, and a longer body as separate values. shortDef holds the self-contained answer; the body handles boundaries, trade-offs, and examples.
Role pages preserve the published salary range in visible copy and emit the same range as baseSalary.
Typed fields give each value a job and let the build reject malformed content.
3. Trustworthy: keep claims honest across representations
Structured data creates another place to overstate reality. Unsupported values stay absent.
Our role pages provide the clearest example. JobPosting appears on individual role pages and stays off the careers index. We leave out directApply because the application continues on an external applicant-tracking system. We do not invent validThrough when a role has no confirmed expiry.
The roles were advertised as remote in the EU, but we did not have a definitive country list when the schema shipped. The collection leaves applicantLocationRequirements absent and emits a build warning. The roles remain ineligible for Google’s remote-job rich result until the real country list exists.
When a role is marked closed or passes validThrough, it leaves the careers index, renders a closed state, and stops emitting JobPosting. Its self-canonical URL stays live.
Trace claims back to their source
Machine-readable mistakes travel easily. Our editorial review treats provenance as part of the implementation.
Evidence-bearing glossary entries carry primary sources. Date-sensitive articles include a “Last verified” date. During a Drupal end-of-life update, we checked every date against Drupal’s documentation. We removed an expected 2028 date because Drupal had not published the release date needed to support it. The article now states the policy and labels the date implied by it as a floor.
A service-page review caught a different problem. Draft copy claimed “40% faster ops,” but the canonical Ingersoll Rand case study did not contain that figure. We cut it.
Every checkable claim needs a source URL, case study, repository file, or measured report.
Preserve the reason behind search decisions
Our migration guides and migration service pages had begun chasing the same query intent. We kept the guides informational and retitled 33 /migrate pages around service intent. The pull request records the before-and-after titles.
Both surfaces remain self-canonical. Their titles and cross-links make their jobs explicit.
Build checks support the editorial review. Content collections validate against their schemas. During the glossary depth pass, all 75 internal links were resolved against the content collection and checked for draft targets. The check was scoped to that change; it was not a site-wide guarantee.
4. Measurable: define the evidence
There is no clean metric for “AI readability.” We watch narrower signals.
Google Search Console is our first-party view of how queries meet pages. In the 24 August report, a query that appeared to contain a leaked assistant prompt placed one of our pages at position 3.8. The content-governance post also appeared for assistant-shaped queries at positions 3.6 to 9.3.
That shows machine-shaped language reaching our pages through Google Search. It does not identify the assistant, prove a fetch, or establish a citation.
We also monitor AI mentions in Prelio and disclose that it is our product. We treat it as a second lens on visibility.
The useful measurements attach to a specific change. We record a glossary entry’s starting queries and position, preserve old and new titles during an intent split, and retest broken URLs after a redirect repair.
For each change, we want four things:
- The condition before the change.
- The repository diff.
- A test of the rendered or live result.
- A defined measurement window.
Without that sequence, a feature inventory can look like an outcome report.
The order we would use on another site
If we were auditing another content-heavy site tomorrow, we would work in this order.
- Fetch representative pages without executing JavaScript and confirm that their substantive content is present.
- Check canonicals, legacy redirects, the sitemap, RSS,
robots.txt, and any edge-layer modifications. - Read the heading outline and inspect whether comparisons, lists, and definitions carry semantic structure.
- Establish stable entity identifiers and make page schemas reference them.
- Generate visible FAQs and their structured representation from one source.
- Separate extractable answers and machine values into typed fields where the content model benefits from them.
- Omit unsupported schema properties and design expiry behavior before content becomes stale.
- Trace statistics, dates, and company claims to canonical evidence.
- Record a baseline and measurement window before expecting a result.
The first three steps expose whether the page survives retrieval and whether infrastructure changes what the application tried to publish.
What this work cannot promise
A machine-readable site can be retrieved and interpreted more reliably. Citation remains outside our control.
llms.txt may remain unread by the systems we care about. A crawler can respect a restrictive policy set by its operator. Search and assistant products can choose another source. Strong markup cannot rescue a thin answer, an unsupported claim, or a page that misses the user’s intent.
Our own implementation still has open work. We found an edge policy contradicting the repository. The canonical decision between our comparison blog posts and their /compare twins was made only as this article went out — the blog versions now point their canonicals at the maintained comparisons, and the consolidation is still inside its measurement window.
Machine readability is a release discipline. The source, build, edge, live response, content model, and measurement need to agree.
On wayf.ai, the work came down to stable identifiers, honest omissions, shared data sources, typed fields, redirect tests, source checks, and live HTTP responses.
FAQ
-
How do I check whether AI systems can read my site?
Fetch representative pages without executing JavaScript and confirm that their main facts, headings, links, and structured data appear in the returned HTML. Then inspect the live
robots.txt, canonicals, redirects, sitemap, and response headers. This tests what a crawler receives, including changes made by a CDN or edge platform. -
Do I need an llms.txt file?
An
llms.txtfile is inexpensive to maintain and can provide a curated map of a site, but major systems do not consistently document support for it. Treat it as an experimental discovery surface and a concise self-description. Keep the sitemap, semantic HTML, internal links, and crawler policy accurate first. -
Does structured data improve AI visibility?
Structured data gives machines explicit information about a page and its entities. It improves consistency and can make a page eligible for search features. It does not guarantee rankings or citations. The visible page, structured data, and underlying evidence should describe the same reality.
Related reading
We're booking content platform
engagements for 2026.
Twenty-five minutes to walk through the work and decide if we're the right team for it. Scoping and a fixed price come after.