Skip to content

fix(seo): publish valid robots.txt crawler rules - #192

Merged
InfernapeXavier merged 1 commit into
masterfrom
fix/robots-txt
Oct 24, 2026
Merged

InfernapeXavier merged 1 commit into
masterfrom
fix/robots-txt

Conversation

@InfernapeXavier

Copy link
Copy Markdown
Owner

Summary

  • Serve a static robots.txt with User-agent: * and Allow: /.
  • Omit a sitemap reference because no sitemap exists and the production sitemap URL returns 404.
  • Leave Cloudflare AI Crawl Control settings unchanged.

Validation

  • Production build, ESLint, and TypeScript checks passed.
  • Local production server returned HTTP 200 and text/plain with the expected directives.
  • Baseline production scan confirmed the missing User-agent failure; recheck production after deployment.

Merge only after the preview deployment succeeds.

@vercel

vercel Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
infernapexavier-github Ready Ready Preview Oct 24, 2026 5:15pm UTC

@InfernapeXavier
InfernapeXavier merged commit ac5cd43 into master Oct 24, 2026
7 checks passed
@InfernapeXavier
InfernapeXavier deleted the fix/robots-txt branch October 24, 2026 17:16

This branch was successfully deployed

1 active deployment
Preview — d68a887d Deployed Oct 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants