fix: SEO indexing, duplicate content, robots.txt, and JSON-LD

[davidmytton]

Search Console reports 681 pages as "Excluded by 'noindex' tag". An audit of the build found that the noindex pages were the wrong ones, and the index was carrying empty duplicates.

Problems

• No Python pages in Google. Every /sdk/:sdk/plus/:variant/ page was noindex and left out of the sitemap. Python has no base SDK route, and /sdk/python/* redirects to /sdk/python/plus/fastapi/*, so none of its 46 pages could be indexed. Express, Hono, and Bun + Hono had no indexable page either.
• Empty copies in the index. The content loader copied every framework-specific page under every SDK and plus-variant prefix, whatever the page's frameworks frontmatter said. 218 copies had no framework content: SlotByFramework found no matching slot, so the page rendered an empty island, with no install command or code, even with JS. 142 of them were indexable and listed in the sitemap, for example /sdk/nuxt/rate-limiting/quick-start/.
• Shared titles and descriptions. 217 indexable pages shared a title with a sibling copy, and 341 shared a meta description.

Changes

• Only copy supported pages. The loader copies a page only to the SDKs listed in its frameworks.
  • The removed URLs are kept as a fixed list, RETIRED_SDK_DOC_ROUTES. Each one 301s to the base SDK's copy for a plus-variant (Express → Node.js), otherwise to the unscoped page.
  • The regenerated vercel.json no longer sends ?f= redirects to removed copies. The unscoped page reads ?f= itself.
  • The SDK switcher only offers SDKs the page supports.
  • Plus-variant sidebars and in-page links fall back to the base SDK's copy.
• Index plus-variant pages. They drop noindex and are added to the sitemap. Unscoped legacy hubs stay noindex.
• Unique titles and descriptions. SDK copies without a framework-specific title get " for " appended. Each framework-specific page gains an sdkDescription template, for example "Quick start guide for adding Arcjet rate limiting to your {sdk} app.", applied to description and og:description. Also fixes a typo and a wrong description on two pages.
• robots.txt. Content-Signal is now search=yes, ai-input=yes, ai-train=yes.
• JSON-LD. The logo, Organization, and SoftwareApplication nodes now match the site-wide graph on arcjet.com. The docs keep softwareHelp, which points at this site. offers is left out because arcjet.com only publishes it on /pricing.

Result

	Before	After
Rendered pages	679	461
Indexable pages	529	436
Plus-variant pages indexable	0 of 125	49 of 49
Pages sharing a title	217	0
Pages sharing a description	341	0

Testing

• npm run build: 0 errors, 0 warnings.
• npm run pw:run: 572 passed. New tests cover supported-copy selection, retired redirects, ?f= handling, switcher filtering, plus-variant indexability, the sitemap, and SDK descriptions.
• A scan of every built page found no links to removed routes.
• Screenshots: only the two /content-moderation/ Linux snapshots changed, because the "Last updated" date grew from Sep 3 to Sep 26. They were regenerated on aarch64, so the playwright-update.yml workflow can confirm them on amd64 if CI disagrees. The macOS snapshots for that page may also be stale for the same reason.

Not in this PR

• Thin pages. /integrations/better-auth/ has no code since its example was removed from arcjet-js (#5633), and /regions/ is one paragraph. Both need content decisions.
• 19 of the edited MDX files fail prettier --check, as they did on main before this change.

After deploy

Resubmit the sitemap in Search Console and click "Validate fix" on the noindex report. Expect about 25 URLs to remain, from the legacy hubs that are noindex on purpose.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
arcjet-docs	Ready	Preview	Sep 26, 2026 9:07pm UTC

[arcjet-review]

Arcjet Review — 🟡 Medium Risk

Decision: Needs Review

Rationale: The PR makes broad but mostly content/routing/SEO-oriented changes: SDK-scoped docs generation, sitemap inclusion, redirect generation, metadata mutation, robots.txt Content-Signal, and JSON-LD. I did not identify hardcoded secrets, user-input injection paths, auth logic changes, or critical security issues. However, the diff is large, includes a generated-looking vercel.json redirect rewrite and binary screenshot baselines, and correctness depends on the fixed RETIRED_SDK_DOC_ROUTES list and redirect ordering being complete. Because I cannot confidently validate all generated redirects and binary snapshots from the diff alone, human review is recommended before merge.

Summary of Changes

Updates docs SEO behavior by only publishing SDK copies for supported frameworks, indexing plus-variant SDK pages, adding per-SDK descriptions/titles, changing robots.txt to allow AI training, aligning JSON-LD entities with the main site, adding retired SDK route redirects, and expanding tests around sitemap, redirects, metadata, SDK switcher behavior, and snapshots.

Escalation Triggers

• Authentication & Authorization: The diff contains the content pattern 'token' in AI/docs metadata text. This appears to refer to AI token budgets rather than authentication tokens, but the configured trigger matches diff content.

Review Focus Areas

• Verify that RETIRED_SDK_DOC_ROUTES is complete and only contains previously published unsupported SDK copies.
  Missing retired routes could create 404s for previously indexed URLs, while incorrect entries could redirect valid pages away from indexable content.
• Check retiredSdkDocRedirectTarget for redirect loops and ensure plus-variant redirects always land on an existing base SDK page or the intended unscoped page.
  This logic controls permanent redirects for many URLs; mistakes would be cached by clients/search engines and may be hard to unwind.
• Confirm entrySupportsFramework and each edited MDX frameworks list produce the intended set of SDK-scoped copies, especially Python plus-variant pages with no base Python SDK route.
  The loader now omits unsupported copies entirely, so incorrect frontmatter could silently remove valid docs pages from the build and sitemap.
• Validate that mutating routeData.entry.data.title and description does not leak modified metadata between SDK copies during dev server reuse or static generation.
  The code appends per-SDK title/description data by mutating route entry data; if entries are reused unexpectedly, titles could accumulate or be attributed to the wrong SDK.
• Review the generated redirect ordering and removed ?f= redirects to make sure Vercel first-match behavior still sends legacy framework query URLs to the correct remaining pages.
  The diff shows a large redirect table rewrite; an ordering or generation error could redirect users/search crawlers to the wrong SDK page.
• Visually confirm the binary screenshot baseline changes, including the corresponding light snapshot.
  Binary snapshot contents cannot be inspected in the textual diff, so human or CI visual verification is needed.

Notes

The diff exceeds the configured 1000-line threshold, primarily because of the large vercel.json redirect changes and fixed retired route list; this reduces confidence in fully automated review. Security review found no secrets, auth implementation changes, injection sinks, cryptography changes, or sensitive data handling issues in the visible diff.

_{Review: b003afff | Model: openai/gpt-5.5 | Powered by Arcjet Review}