# TableJourney
# Policy: open to all well-behaved crawlers, including AI retrieval and
# training bots. Editorial referrals from AI search (ChatGPT, Perplexity,
# Claude, Google AI Overviews, etc.) are an explicit growth channel.
#
# No Content-Signal line (removed 2026-10-06, owner's call): an all-yes
# signal grants only what Allow: / already grants, nothing enforces it,
# and Lighthouse and Bing's tester flag the unknown line as an error
# (Lighthouse SEO 92 instead of 100 on every page). Add it back only to
# say no to something, e.g. ai-train=no.
#
# AI-optimized manifests: /llms.txt (site map for agents) and
# /llms-full.txt (per-entity deep manifest with citations). Those two
# also carry the agent API: the OpenAPI spec at /api/v1/openapi.json,
# interactive docs at /api/v1/docs, and the MCP server at /mcp with its
# manifest at /.well-known/mcp.json. All public, no key needed.
User-agent: *
Allow: /
# /api/v1/docs is the one thing under /api/ that is a real page rather
# than JSON: Swagger UI, 200 text/html, worth indexing. Longest-match
# wins over the Disallow below for every crawler that honours Allow.
Allow: /api/v1/docs
Disallow: /search?
Disallow: /api/
# /mcp speaks streamable HTTP and does not answer a plain GET; a request
# to it hangs until the client gives up. There is nothing for a crawler
# to index and a stalled fetch costs crawl budget, so keep them out.
# Agents find it through llms.txt and /.well-known/mcp.json instead.
Disallow: /mcp

Sitemap: https://tablejourney.com/sitemap.xml