robots.txt was written for crawlers that traverse the web unattended, not for an agent fetching one page because a human asked. Cloudflare's new Search, Training and Agent controls finally let site owners say which one they mean.

Claude's built-in web fetch bows to robots.txt and can't run JavaScript. A self-hosted Playwright MCP server behind a free Cloudflare MCP portal fixes both.