Robots.txt Tester & Generator
Paste your robots.txt, enter a URL, and see whether a crawler can reach it — along with the exact rule that decided, and every rule it beat. Search Console's tester was retired in 2023 and the report that replaced it can't answer this question, so it runs here instead, in your browser.
Your robots.txt
Paste the file from yoursite.com/robots.txt.
199 bytes · 3 groups · 5 rules · 1 sitemap
Nothing is uploaded and nothing is fetched — the file is read and matched in your browser, so a staging file with paths you haven’t shipped stays on your machine.
Test a URL
Crawlable
/cart
- Group:
User-agent: googlebot— line 6.- Decided by:
- No rule matches this URL, and anything not disallowed is crawlable.
This crawler has a group of its own, so the rules under User-agent: * do not apply to it. Only the most specific group is read — the rest are ignored, not combined.
What’s in the file
Every line parses. That means the file says what it looks like it says — not that the rules are the right ones.
A robots.txt reports none of this in production. An unrecognised line is skipped in silence, so a file full of directives that do nothing looks exactly like one that works.
Why this tool exists
Search Console used to have a robots.txt Tester: you typed a URL, picked a crawler, and it told you which line blocked it. Google retired it in 2023 and replaced it with a robots.txt report that shows which files Google fetched, when, and whether they parsed — useful, and a different question entirely. For testing a specific URL, the help page points you at URL Inspection, which works only on a property you have verified and only for a URL that already exists.
So the two moments when you most need an answer are exactly the two it can’t cover: beforeyou ship a rule, and on a site that isn’t yours. That is the gap this page fills.
The four rules that decide everything
Google publishes how it interprets robots.txt, and the tool above implements it exactly. Four rules account for almost every surprise.
1. The longest matching path wins — not the first
Rules are not read top to bottom like firewall rules. Every rule that matches the URL is collected, and the one with the longest path decides. Allow: /p beats Disallow: / for /page because two characters beat one, whichever order they sit in. When two matching rules are the same length, Google uses the least restrictive, so the allow wins — which is exactly how Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php keeps that one file reachable.
2. Only the most specific group applies
This is the one that quietly breaks real sites. A crawler finds the single group whose user-agent best matches its name and ignores every other group, including the catch-all. Add three lines for User-agent: Googlebot to a file that already has a long User-agent: * section, and Googlebot now obeys three rules instead of twenty. Nothing warns you; the file just got longer and stopped working.
The grouping is positional, too. Consecutive user-agent lines share one group, but a user-agent line that appears after a rule starts a new one — so this file gives Bingbot no rules at all:
User-agent: GooglebotDisallow: /private/User-agent: Bingbot
3. Blocking is not hiding
Disallow stops Google crawling a URL. It does not stop Google indexing it. A blocked page that other pages link to can still show up in results — typically as a bare URL with no description, because Google knows it exists and was never allowed to look at it.
To keep a page out of the index you need a noindex meta tag or X-Robots-Tag header, which Google can only see if it is allowed to crawl the page. Blocking it in robots.txt guarantees the tag is never read. And Noindex: as a robots.txt directive stopped working in 2019 — it is still in a surprising number of live files, doing nothing.
4. One file, one host
A robots.txt governs the exact host, protocol and port it is served from. blog.example.com needs its own file; example.com and www.example.com are different hosts; and paths are case-sensitive, so Disallow: /private/ leaves /Private/ wide open. Google reads the first 500 KiB and ignores whatever follows.
What about AI crawlers?
The builder above can add blocking groups for the crawlers behind ChatGPT, Claude, Perplexity, Gemini and Common Crawl. Two things are worth being clear about before you use it.
robots.txt is voluntary.It is a sign, not a lock. The major operators publish their tokens and honour the file, and anyone who doesn’t is unaffected by anything you write in it.
Blocking is a visibility decision, not only a privacy one. Some of these crawlers train models; others build the index behind an answer engine that cites sources. Blocking the second kind removes you from answers people are already reading instead of clicking. The case most often got wrong is Google-Extended: it governs Gemini and Vertex AI, and does not take you out of AI Overviews, which are served through ordinary Googlebot. If being quoted by assistants is something you care about, our guide to ranking in AI Overviews covers what actually influences it, and AI Visibilitytracks whether you’re being mentioned at all.
After the file is right
A correct robots.txt only proves that one rule does what you meant. It says nothing about the pages behind it — whether they redirect, carry a stray noindex, sit four clicks from the homepage, or link to something that 404s. Those are site-wide questions and they need a crawl, which is what Site Audit is for; the audit issues worth fixing first starts with exactly this class of problem, because a page a crawler can’t reach is the only SEO issue that makes every other one irrelevant.
Questions
- How do I test whether a URL is blocked by robots.txt?
- Paste your robots.txt into the tool above, enter the URL and pick the crawler. It applies Google's own matching rules — longest matching path wins, and an allow beats a disallow of the same length — and shows you the exact line that decided it. Nothing is uploaded; the matching happens in your browser.
- Where did the Search Console robots.txt tester go?
- Google retired it in 2023 and replaced it with a robots.txt report. The report shows which robots.txt files Google found for your site, when they were last fetched and any parse errors — but it does not let you test a URL against your rules. Search Console's help page points you at the URL Inspection tool for that, which only works on a property you have verified, and only for URLs that already exist.
- Does robots.txt stop a page from appearing in Google?
- No, and this is the most expensive misunderstanding in SEO. Disallow stops Google crawling a page, not indexing it — a blocked URL that other pages link to can still appear in results, usually with no description, because Google knows the URL exists but was never allowed to look at it. To keep a page out of the index you have to let it be crawled and serve a noindex meta tag or X-Robots-Tag header. Blocking it in robots.txt guarantees Google never sees that tag.
- Which rule wins when two rules match?
- The one with the longer path, regardless of the order they appear in the file. If both are the same length, the least restrictive wins, which means an allow beats a disallow. That is why Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php leaves that one file crawlable.
- Do the rules under User-agent: * apply to every crawler?
- Only to crawlers that don't have a group of their own. A crawler reads the single most specific group that names it and ignores the rest, so the moment you add a User-agent: Googlebot group, none of your catch-all rules apply to Googlebot any more. It is a silent change: the file looks longer and does less.
- Are wildcards allowed in robots.txt?
- Google supports two: * matches any run of characters, and $ marks the end of the URL. They are not part of the original standard, so other crawlers may treat them as literal characters. A trailing * does nothing, because matching is already a prefix match — /fish and /fish* are identical.
- Is robots.txt case-sensitive?
- The field names and user-agent values aren't, but paths are. Disallow: /private/ does not block /Private/. The file itself must also be named robots.txt in lower case and live at the root of the domain.
- Does one robots.txt cover my subdomains?
- No. A robots.txt governs only the exact host, protocol and port it is served from, so blog.example.com needs its own file, and so does the https version if you still serve http. This catches people out after a migration more than anything else.
- Can I block ChatGPT and other AI crawlers here?
- You can ask them to stay away, and the major ones do publish user-agent tokens and honour the file. But robots.txt is voluntary — it is a sign, not a lock — and blocking a crawler that feeds an answer engine is a visibility decision, not just a privacy one. Blocking Google-Extended is the case worth understanding: it covers Gemini and Vertex AI, and does not affect Google Search or AI Overviews, which are crawled by regular Googlebot.
- Is this tool free?
- Yes — no signup, no limit, and nothing leaves your browser. The file you paste is parsed and matched locally; we never fetch your site.
More free tools
- SERP Snippet PreviewSee where Google cuts your title and meta description — measured in pixels, not characters.
- Hreflang Tag GeneratorBuild the tags for every language version of a page, with the codes checked against the standards Google actually reads.
- Keyword Clustering ToolTurn a keyword export into groups you could write a page for — in your browser, with the list never leaving it.
- Striking Distance Keyword FinderDrop in a Search Console export and see the queries sitting just off page one, sorted by what they're already earning.
- Keyword Cannibalization CheckerFind the queries where two of your own pages are competing, and see how the impressions are split between them.
- CTR Curve CalculatorBuild a click-through-rate curve from your own Search Console data instead of trusting someone else's study.
One URL is a spot check. What about the other 4,000?
A single blocked directory can hide a whole section of a site from search. Site Audit crawls every page you have and reports what is blocked, redirected, noindexed or unreachable — sorted by how much it costs you.