robots.txt was written for crawlers that traverse the web unattended, not for an agent fetching one page because a human asked. Cloudflare's new Search, Training and Agent controls finally let site owners say which one they mean.
A talented engineer with agents now ships more than a hundred of them. Brooks' surgical team was always the better design; AI just made it affordable, and headcount stopped being a proxy for output.
Claude's built-in web fetch bows to robots.txt and can't run JavaScript. A self-hosted Playwright MCP server behind a free Cloudflare MCP portal fixes both.