From 89700002be2817aa35ba2bd328a56e444cca0210 Mon Sep 17 00:00:00 2001 From: mpot37 Date: Fri, 14 Aug 2026 16:53:04 +0200 Subject: [PATCH 1/2] fix: add custom robots.txt with explicit AI-crawler Allow rules Mintlify auto-generates robots.txt but doesn't include explicit per-bot Allow lines for AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) -- flagged severe in the check.outrun.at GEO audit and tracked as Linear REQ-271. Confirmed with Mintlify support that a project-root robots.txt overrides the auto-generated one, but is served as-is (loses the default Content-Signal directive) -- so this file reproduces every directive from the current auto-generated robots.txt (Content-Signal, the /cdn-cgi/ and /_next/ rules, the sitemap) and adds explicit Allow: / blocks for each AI crawler on top. --- robots.txt | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 robots.txt diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..9642e77 --- /dev/null +++ b/robots.txt @@ -0,0 +1,25 @@ +User-agent: * +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ + +User-agent: GPTBot +Allow: / + +User-agent: OAI-SearchBot +Allow: / + +User-agent: ClaudeBot +Allow: / + +User-agent: anthropic-ai +Allow: / + +User-agent: PerplexityBot +Allow: / + +User-agent: Google-Extended +Allow: / + +Sitemap: https://docs.request.network/sitemap.xml From e16b37f154d627728d10425d2462ba7c0ff374fd Mon Sep 17 00:00:00 2001 From: mpot37 Date: Mon, 17 Aug 2026 09:23:19 +0200 Subject: [PATCH 2/2] fix: repeat wildcard exclusions in named crawler groups Per robots.txt group-matching rules, a crawler follows only the most specific User-agent group that names it, not the wildcard group as well. The named AI-crawler groups only had "Allow: /", so they were bypassing the Content-Signal directive and the /cdn-cgi/ and /_next/ exclusions defined under User-agent: *. Addresses Greptile review feedback on PR #119. --- robots.txt | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/robots.txt b/robots.txt index 9642e77..a2cdfea 100644 --- a/robots.txt +++ b/robots.txt @@ -5,21 +5,45 @@ Allow: /_next/image Disallow: /_next/ User-agent: GPTBot +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ Allow: / User-agent: OAI-SearchBot +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ Allow: / User-agent: ClaudeBot +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ Allow: / User-agent: anthropic-ai +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ Allow: / User-agent: PerplexityBot +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ Allow: / User-agent: Google-Extended +Content-Signal: ai-train=yes, search=yes, ai-input=yes +Disallow: /cdn-cgi/ +Allow: /_next/image +Disallow: /_next/ Allow: / Sitemap: https://docs.request.network/sitemap.xml