Robots Txt Tester Online: Check Any URL Against Your Robots.txt in Seconds
CodexFuse Team · 2026-11-05T07:04:32.214Z · 9 min read min read
Test any URL against your robots.txt rules in seconds — allowed or blocked, with the exact rule that decided it. Free, private, and no sign-up.
A robots txt tester answers one question in seconds: for a given crawler, is this URL allowed or blocked by your robots.txt file? You paste your rules, pick a user-agent such as Googlebot, enter a path like /admin/dashboard, and the tester returns ALLOWED or BLOCKED — along with the exact rule that decided it.
One misplaced Disallow can hide an entire shop from Google for weeks; one missing rule can let search engines index an admin login page. Most site owners never test their robots.txt — they write it once, assume it works, and discover the damage only when traffic falls.
What Does a Robots.txt Tester Actually Check?
A robots.txt tester checks whether a specific URL path is allowed or blocked for a specific crawler under your robots.txt rules. It applies the same longest-match precedence real crawlers use and names the exact Allow or Disallow rule that decided the verdict — nothing more, nothing less.
Matching is by prefix: a rule matches any path starting with its text. Rules are grouped by user-agent, so Googlebot can be treated differently, with the * group as the fallback. When several rules match one path, the longest match wins — rule order does not matter.
Test your robots.txt on a staging copy of the rules before you publish them. The CodexFuse tester runs entirely in your browser and fetches nothing, so you can safely paste rules you have not deployed yet.
A Real Example: How One Line Hid 412 Product Pages
Take a fictional scenario: Meadow & Main, a small home-goods store on WooCommerce. The owner wanted Google out of the cart and checkout pages, so a developer added one line: Disallow: /shop/
Every product URL lived under /shop/ — /shop/canvas-tent, /shop/linen-throw, all 412 of them. Within three weeks, organic sessions fell from around 1,900 a week to roughly 640, and the best-selling tent vanished from Google.
A tester would have caught it in thirty seconds: paste the rules, set the user-agent to *, test /shop/canvas-tent — BLOCKED by Disallow: /shop/. The fix took two minutes: replace the broad rule with Disallow: /shop/cart and Disallow: /shop/checkout, then re-test. /shop/canvas-tent reads ALLOWED, /shop/cart stays BLOCKED, and after a re-crawl request the product pages returned within ten days. The lesson: untested robots.txt is dangerous.
How to Test Your Robots.txt in Four Steps
Use the free robots.txt tester — four fields, in order.
1. Paste your rules into the first box
Copy the full file — every User-agent group, every Allow and Disallow line. Partial rules give partial answers.
2. Enter the crawler user-agent
Use * for all crawlers, or a specific one such as Googlebot. A crawler with no group falls back to *.
3. Enter the URL path to test
Type the path exactly as it appears on your site, for example /admin/dashboard. A full URL works too — only the path is tested.
4. Read the verdict and the rule that matched
You get ALLOWED or BLOCKED, plus the exact rule that decided it. With the sample rules Disallow: /admin/, Allow: /public/ and Disallow: /private: /admin/dashboard is BLOCKED, /public/page is ALLOWED, and /about is ALLOWED because no rule matches it. Copy the result with one click.
Test the paths you care about most, not just the ones you changed: homepage, a product page, login page, and cart. Five quick tests catch most accidents.
Which Rule Wins When Allow and Disallow Both Match?
The longest matching path wins — not the first rule listed. If Disallow: /admin/ and Allow: /admin/public/ both match /admin/public/page, the page is ALLOWED because Allow: /admin/public/ is the longer match. Order in the file does not matter; only the length of the matching path decides the verdict.
This is the most useful fact about robots.txt: you can block a whole section and carve out exceptions — block /admin/, allow /admin/public/. One edge case: matching is prefix-only, so Disallow: /admin/ blocks /admin/dashboard but not the bare path /admin. Test both forms.
Five Robots.txt Mistakes That Quietly Cost You Traffic
- Blocking a section to hide one page: a broad Disallow such as /shop/ blocks everything underneath it. Write the narrowest rule, then test a path you meant to keep.
- Forgetting the trailing slash: Disallow: /admin/ and Disallow: /admin are different rules. If you mean the directory, include the slash — and test the bare path too.
- Treating * as a wildcard: in prefix matching, Disallow: /private/* does not block /private/x — the asterisk is literal. Use Disallow: /private/ instead.
- Assuming Disallow removes pages from results: it does not — it stops crawling, and already-indexed pages can stay visible. To remove a page, use noindex or delete it.
- Editing live without testing: robots.txt has no undo button. A typo can block your whole site before you notice. Paste first, test, then publish.
A blocked page is invisible to search, and an untested robots.txt is how pages get blocked by accident. Move your site to CodexFuse and our fully managed hosting keeps your crawl setup, security and backups handled for you — on pay-as-you-go credits, with no subscriptions.
Get Started with CodexFuseTester, Validator, or Search Console — Which Do You Need?
| Tool | What it answers | When to use it |
|---|---|---|
| Robots.txt tester | Is this URL allowed or blocked for this crawler? | Before publishing any robots.txt change |
| Robots.txt validator | Is the file itself well formed? | When you suspect syntax errors or missing groups |
| Search Console report | What did Google fetch, and what is it blocking? | After publishing, to confirm live behaviour |
The two tools complement each other. The robots.txt tester checks individual URLs against pasted rules; a validator scores the file itself — CodexFuse's robots.txt validator checks four things (User-agent line, rules present, wildcard group, no syntax errors) out of 100, with 75 or above reading as VALID. Run the validator when you write the file, the tester for specific pages, and Search Console afterwards. The canonical URL checker catches a different indexing mistake, and the Google index checker confirms whether a page is in the index once crawling is fixed.
What a Robots.txt Tester Cannot Do
A tool trusted beyond its scope creates the same false confidence as no tool at all.
- It does not fetch your live file: you paste the rules and the check runs in your browser. It cannot tell you whether the file on your server matches what you pasted — always copy from the live file.
- It does not validate syntax: a malformed line may be silently ignored by crawlers. Pair the tester with a validator before publishing.
- It does not check indexation: ALLOWED means Google may crawl the page, not that it has indexed it or will rank it.
- It does not remove indexed URLs: adding Disallow after a page is indexed will not take it out of results. Use noindex or delete the page.
- It matches by prefix only: wildcards are literal, so Disallow: /private/* behaves differently from what many expect.
Building robots.txt from scratch? Start with the robots.txt generator, then verify every rule with the tester above. For larger sites, the HTML sitemap generator helps crawlers find the pages you do want indexed.
From a Fixed Robots.txt to a Site That Stays Fixed
Fixing one robots.txt file is a single afternoon. Keeping a site healthy is ongoing work: hosting that stays up, certificates that renew, backups that exist, patches applied on time. Most small teams never have time for it — which is why the same avoidable mistakes recur.
This is where CodexFuse fits as a concrete workflow. Move your site to CodexFuse's fully managed hosting and the platform handles the server side — hosting, security monitoring and backups without you babysitting a server. You pay with credits, pay-as-you-go: no monthly subscription, no lock-in. After migrating, run the robots.txt tester once against your live rules, then re-test after every site change.
Frequently Asked Questions
What does a robots.txt tester do?
It answers one question: is this URL allowed or blocked for this crawler? Paste your rules, pick a user-agent, enter a URL path, and it returns ALLOWED or BLOCKED plus the exact rule that decided it.
How do I test if a URL is blocked by robots.txt?
Paste your rules into the tester, set the user-agent to * and enter the path. With the sample rules, /admin/dashboard is BLOCKED, /public/page is ALLOWED, and /about is ALLOWED because no rule matches it.
Which rule wins when Allow and Disallow both match?
The longest matching path wins, not the first rule listed. With Disallow: /admin/ and Allow: /admin/public/, /admin/public/page is ALLOWED — the longer match — while /admin/dashboard stays BLOCKED.
Does Disallow: /admin/ block /admin?
No. Matching is by prefix, so Disallow: /admin/ only matches paths starting with /admin/, trailing slash included. The bare path /admin does not match, so it is ALLOWED. The same applies to Allow: /public/ and /public.
Is a robots.txt tester the same as a robots.txt validator?
No. The tester checks one URL against pasted rules; a validator checks the file itself. CodexFuse's validator scores four checks (User-agent, rules, wildcard group, no syntax errors) out of 100, with 75 or above reading as VALID.
Does the tester fetch my live robots.txt?
No. You paste the rules and the check runs entirely in your browser — nothing is fetched, so staging files and unpublished rules are safe to test. A full URL works as the path input; only its pathname is used.
Test Your Robots.txt Before Your Next Deploy
Your robots.txt file is a few lines of text with the power to hide your entire site from Google — or to leave your private pages exposed. Open the free robots.txt tester now, paste your live rules, and check the five paths that matter most: your homepage, a product page, your login page, your cart, and your checkout. It takes two minutes.
Then fix the bigger problem underneath: a site you have to babysit. Move your hosting, security and backups to CodexFuse today — fully managed, on pay-as-you-go credits, no subscriptions, no lock-in. Get started with CodexFuse now and make your next deploy a tested one.
