# ============================================================
# StageRage — robots.txt
# Served at https://stagerage.online/robots.txt (MUST return HTTP 200)
# Last reviewed: 2026-09-24
#
# ⚠️  DO NOT add "Disallow: /api/" here.
#     StageRage is a client-rendered SPA: every public page (venues, reviews,
#     events) builds itself in the browser by fetching /api/entities/... via
#     XHR. Googlebot obeys robots.txt for those fetches too — so blocking /api/
#     stops Googlebot from loading page data, and every venue page renders as
#     "Venue Not Found" → soft-404 → zero indexation. (This exact block caused
#     StageRage's soft-404s in July 2026.)
#     To keep raw API responses OUT of the index without blocking the render,
#     send an "X-Robots-Tag: noindex" HTTP header on /api/ responses instead
#     (see Nginx/Express note kept with this file). robots.txt controls CRAWL;
#     X-Robots-Tag controls INDEX. Use the header, never a Disallow, for /api/.
#
# POLICY SUMMARY
#   • Search engines, AI answer/search bots, and AI training bots may crawl
#     public StageRage pages and the public API data required to render them.
#   • Account, admin, personal, private-tour, operational-editor, and
#     multiplayer-session routes remain disallowed for every well-behaved
#     crawler.
#   • API responses also send X-Robots-Tag: noindex, nofollow: crawlers may
#     fetch public API data to render a page, but should not index raw JSON.
#   • Public crew profiles (/user/<id>) stay CRAWLABLE on purpose: they carry
#     a rendered "noindex, follow" meta. Blocking the URL would hide that tag
#     and produce URL-only search listings instead (robots.txt controls CRAWL,
#     the meta controls INDEX — same rule as /api/ above).
# ============================================================

# ---- AI crawlers: deliberately NO named User-agent groups ----
# Every compliant AI crawler (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot,
# Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User,
# Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher,
# Amazonbot, Bytespider) reads the single User-agent: * group below. A named
# group would NOT inherit these Disallow rules — adding "Allow: /" for one bot
# alone would expose /admin and every private route to exactly that bot. If a
# named group ever becomes necessary, duplicate every rule into it verbatim.

# ---- All well-behaved crawlers, including AI training crawlers ----
User-agent: *
Allow: /

# Auth & account flows — no SEO value
Disallow: /login
Disallow: /register
Disallow: /forgot-password
Disallow: /reset-password
Disallow: /verify-email
Disallow: /oauth-callback
Disallow: /splash

# Private / personal pages
Disallow: /profile
Disallow: /notifications
Disallow: /privacy-settings
Disallow: /venue-dashboard
Disallow: /tour-dashboard
Disallow: /event-dashboard
Disallow: /review-drafts
Disallow: /venue-import
Disallow: /lobby
Disallow: /tour-plan/
Disallow: /coordinators
# "/toolkit" without a trailing slash covers the Toolkit index itself as well
# as every "/toolkit/..." route (the previous "/toolkit/" rule missed /toolkit).
Disallow: /toolkit
Disallow: /tour-planner-ai
Disallow: /venue-assistant
Disallow: /venue-assistant-agent
Disallow: /friends
Disallow: /my-uploads
Disallow: /loader-helper
Disallow: /stage-plot

# Multiplayer session URLs — live rooms and invite links are private state.
# "*" is the standard wildcard (Google syntax): the public /games pages and
# the static /games/<title> pages listed in the sitemap stay crawlable.
Disallow: /games/*/join/
Disallow: /games/*/lobby

# Action / conversion pages (no indexable content)
Disallow: /review/new
Disallow: /upgrade

# Admin
Disallow: /admin

# NOTE: /api/ is intentionally NOT disallowed — see the warning at the top.
# NOTE: /user/<id> is intentionally NOT disallowed — see the policy summary.

# ---- Sitemap ----
Sitemap: https://stagerage.online/sitemap.xml