— Free SEO tool
robots.txt checker and tester
One wrong line in robots.txt can stop Google crawling the whole site. This reads the file the way Google does and tells you what it blocks.
Free tool · robots.txt checker
Check your robots.txt and test any page
Put in your site’s address. The checker reads robots.txt, flags rules that block too much, and lets you test any page against it for Googlebot, Bingbot and others.
Crawling problems are usually one of several. The free audit checks the whole site and says what to fix first.
Get a free SEO audit01 How it works
What it checks
Whole-site blocks
A Disallow: / left over from a staging site is the classic launch mistake. It’s flagged first.
Blocked scripts and styles
Google renders pages like a browser. If robots.txt blocks the CSS or JavaScript, it can’t see the page as visitors do.
Rules Google ignores
Crawl-delay and Noindex in robots.txt aren’t supported by Google. The checker points them out so you know they’re doing nothing there.
The tester
Type any path and pick a crawler. It applies Google’s matching: the most specific group for that crawler, then the longest matching rule, with Allow winning a tie.
Sources: Google: How Google interprets robots.txt · Google: Introduction to robots.txt
02 Questions
What people ask
Does blocking a page in robots.txt keep it out of Google?
Not reliably. It stops crawling, not indexing: a blocked page can still appear if other sites link to it. To keep a page out, allow crawling and use a noindex tag.
Do I need a robots.txt file?
No. Without one, Google crawls everything it can find. It’s useful for pointing to your sitemap and keeping crawlers out of things like search result pages.
How big can robots.txt be?
Google reads the first 500 KiB and ignores the rest. Most files are a few lines.
— Next step
Start with a free SEO audit
Send me your site and you’ll get a short, honest read on what’s wrong. If you want the detail after that, the full audit is a paid piece of work, or part of a retainer.
- 01The main things holding your site back, in plain English
- 02Which of them are worth fixing first
- 03Whether a full audit or a retainer makes sense, or neither