Robots.txt Generator
Generate a robots.txt with sensible presets, custom rules, a sitemap line, and a copyable crawler policy pack that pairs robots.txt with llms.txt and a starter deployment check.
Runs in your browser
Nothing uploaded
Free, no signup
Ctrl + Enter to run · Recent brings back your last inputs · Everything stays in this browser
Robots.txt Generator: questions
Can robots.txt block every AI system?
No. robots.txt is a public crawl directive that cooperative crawlers may honor. It is not access control. Use authentication or server rules for private content.
What is in the crawler policy pack?
The pack includes the generated robots.txt, publication paths, llms.txt and sitemap reminders, review notes, and a starter Node check for the live files.
Should I add an llms.txt file too?
Use robots.txt for crawl directives and llms.txt as a public Markdown guide to important content. The llms.txt convention is proposed and useful for discovery, not a guaranteed ranking signal.
Made something with AI? Share it as a private link.
Drop in the HTML from Claude, ChatGPT or any editor and get a link your client can open anywhere — with a password, view tracking, comments and approval.