Robots.txt generator
Pick what crawlers can reach, add your sitemap, and copy a robots.txt file that’s ready to upload.
- Free
- No sign-up
- Nothing stored
Free tool · robots.txt generator
Write a robots.txt file
Choose what crawlers can reach. Upload the result to the root of your site as /robots.txt.
robots.txt
Runs in your browser. Nothing you type is stored or sent anywhere.
What robots.txt does and doesn’t do
It controls crawling
The file tells crawlers which addresses they may request. It lives at the root of the site, at /robots.txt, and nowhere else.
It doesn’t hide pages
A blocked page can still appear in Google if other sites link to it. To keep a page out of results, use a noindex tag and leave the page crawlable so Google can see the tag.
Most small sites need very little
If nothing needs hiding from crawlers, an “allow everything” file with the sitemap line is enough. The common mistake is blocking too much.
Sources: Google: Introduction to robots.txt · Google: How to write and submit a robots.txt file · Google: Overview of Google crawlers
02 Questions
What people ask
Does blocking AI crawlers affect my Google rankings?
No. Google-Extended only controls whether your content is used for Google’s AI models; it’s separate from Googlebot, which handles search. Whether a crawler obeys the file is up to the crawler.
How do I check my current file?
Use the robots.txt checker: it reads the live file and tests any page against it.
Where do I upload it?
To the top level of the site, so it loads at yourwebsite.com/robots.txt. Shopify, Wix and Squarespace manage the file for you and offer limited editing.
More free tools
— Next step
Start with a free SEO audit
Send me your site and you’ll get a short, honest read on what’s wrong. If you want the detail after that, the full audit is a paid piece of work, or part of a monthly plan.
- 01The main things holding your site back, in plain English
- 02Which of them are worth fixing first
- 03Whether a full audit or a monthly plan makes sense, or neither