Tool AT-W48REV. 2026-A

robots.txt tester: can Googlebot read this page?

Whether Googlebot may fetch a page is decided by a single line in robots.txt: the longest matching rule in the group that applies to that crawler. Paste your file and enter paths or full URLs. For Googlebot, Bingbot, GPTBot and other crawlers, the tester shows whether a path is allowed or blocked and which line decides it. It also flags anything Google reads differently from the standard.

  • free
  • no sign-up
  • Text stays in the browser

Paste your robots.txt and check path by path

Tester · robots.txt according to RFC 9309

The evaluation follows RFC 9309, with the documented quirks of individual crawlers. Whether a crawler abides by it is up to its operator.

Open your-company.de/robots.txt in your browser and copy the whole text. You can fetch the file in the AI crawler check.

Up to 50. For full URLs, the path and parameters count, for example /shop?sort=preis.

For which crawlers?
Search engines · 6
More Google crawlers · 3
AI crawlers · 24

The name the operator gives for robots.txt (product token), not the full browser identifier.

The tester calculates in your browser. Text and URLs do not leave your device.

Result

Paste your robots.txt and enter paths. There is an example to try out.

How a crawler decides whether to fetch a page

Since September 2022 the procedure has been laid down as RFC 9309; before that it was an agreement from 1994, which the big crawlers followed more or less closely Sun et al. 2010. The tester calculates according to the RFC, in this order:

  1. The file. It sits at /robots.txt in the root directory and applies only to exactly that host, with protocol and port. The file on your-company.de does not apply to www.your-company.de or to shop.your-company.de.
  2. The group. The crawler looks for the group with its name, upper and lower case ignored. Several groups with the same name are merged. If there is none, “User-agent: *” applies, and if that is missing too, everything is allowed.
  3. The rule. Within the group, the longest Allow or Disallow line that matches the start of the path wins. If they are equally long, Allow wins. The order of the lines makes no difference.
  4. The characters. An asterisk stands for any number of characters, a dollar sign at the end for the end of the URL. Upper and lower case in the path count. The crawler compares umlauts in percent-encoding, so “/über” and “/%C3%BCber” are the same.
  5. The exception. Anyone may fetch the robots.txt itself, and Sitemap lines apply regardless of the groups.
WHICH GROUP APPLIES?CRAWLER/robots.txtOWNGROUPELSE GROUP*ELSE ALLALLOWEDWHICH RULE APPLIES?PATHMATCHING RULESLONGESTWINS, EQUALLENGTH: ALLOWALLOWED ORLOCKEDWHICH GROUP APPLIES?CRAWLER/robots.txtOWN GROUPELSE GROUP *ELSE ALL ALLOWEDWHICH RULE APPLIES?PATHMATCHING RULESLONGEST WINS, TIELENGTH: ALLOWALLOWED OR BLOCKED
FIG. 1 · HOW A CRAWLER READS ROBOTS.TXTGOLD = YOUR FILE · TWO QUESTIONS, ANSWERED IN TURN

What a pattern matches

Examples, calculated with the same function as the tester
Patternmatchesdoes not match
/fisch/fisch, /fisch.html, /fischkopf/rezept/Fisch, /katalog/fisch
/fisch//fisch/, /fisch/?sorte=lachs/fisch, /fisch.html
/*.php/index.php, /ordner/seite.php?id=1/, /seite.PHP
/*.php$/seite.php/seite.php?id=1, /seite.php5
/*?s=/?s=preise, /blog/?s=test/blog/?seite=2

The comparison always starts at the beginning of the path, which is why /fisch does not match /katalog/fisch. If you mean “anywhere”, put an asterisk in front.

The longest rule wins

In the example, the group blocks the shop but makes an exception for the offers. For /shop/angebote/herbst the tester calculates: allowed by line 3 “Allow: /shop/angebote/” in the group “User-agent: *”. At 15 characters it is the longest of 2 matching rules. For /shop/kasse the result is blocked, because only the shorter rule matches.

For a long time, commercial crawlers repeatedly violated or misunderstood certain rules Giles et al. 2010. Since the RFC the calculation is unambiguous, but not every crawler follows it. That is why the tester flags lines that crawlers read differently.

User-agent: *
Disallow: /shop/
Allow: /shop/angebote/

The most common trap: groups inherit nothing

Once a crawler gets a group of its own, the catch-all group no longer applies to it, not even in part. In the example, /intern/ is to be blocked for everyone, but Googlebot gets its own group for drafts. The result for /intern/preise: Bingbot is blocked, Googlebot is allowed. Whatever is meant to apply to everyone therefore belongs in every named group.

Named groups are common. In an early survey of 2,925 robots.txt files, the large search engines mostly got more access than others through groups of their own Sun et al. 2007b; a later survey of about 6 million websites found no notable preference Kolay et al. 2008. Whether intended or not, the tester shows the group that applies for each chosen crawler and, in the overview, which crawlers each group applies to.

User-agent: *
Disallow: /intern/

User-agent: Googlebot
Disallow: /entwurf/

Where Google reads robots.txt differently from the standard

Google helped write the RFC and follows it at its core. In a few places Google’s parser is more lenient or stricter, and its documentation settles what the RFC leaves open. The tester calculates according to the RFC and reports each of these places as a note, so that you end up with a file that works the same for every crawler.

RFC 9309 compared with Google, Google documentation as of 23 October 2026
PointRFC 9309Google
File sizeread at least 500 KiBat most 500 KiB, the rest is ignored
Cacheat most 24 hours, unless the file is unreachableusually up to 24 hours, longer if refreshing fails
Redirectfollow at least fiveat least five, then like 404
404 and other 4xxno robots.txt, everything allowedthe same, except 429
5xx or unreachableeverything blocked; after a longer time (for example 30 days) treated as missing, or the stored version12 hours without crawling, then up to 30 days the last good version (without one, no restriction), afterwards as if there were no file, if the site is otherwise reachable
Directivesuser-agent, allow, disallow; a crawler may interpret other lines, such as sitemaponly user-agent, allow, disallow and sitemap; noindex, nofollow and crawl-delay have had no effect since 1 September 2019
Typos, missing colonline does not countGoogle’s parser still reads some typos such as “Disalow” and lines without a colon
Name with suffixnames consist only of letters, - and _“googlebot/1.2” and “googlebot*” count as “googlebot”
Umlauts, special characterspercent-encode before comparingcompares in percent-encoding
Sitemapa crawler may interpret itfull URL with protocol and host, may be on a different host

Sources: RFC 9309, Google on robots.txt (as of 23 October 2026), Google blog of 2 July 2019, Google’s parser. Checked on 14 November 2026.

Other search engines have quirks of their own. Bing respects Crawl-delay and, without its own bingbot group, reads a msnbot group; Yandex has not counted Crawl-delay since 22 February 2018 and has a group for all its robots with “User-agent: Yandex”. Without a group of its own, Apple’s crawler follows the Googlebot group, and Google’s advertising crawlers skip the catch-all group entirely. The table below states which of these behaviours the tester accounts for.

Which names belong in robots.txt

The User-agent line takes the product token, that is, the short name the operator gives for robots.txt: “Googlebot”, not the full identifier that appears in the server log. The list in the tester contains only names that their operator documents, and calculates special cases only where the operator describes them.

Search engines and Google crawlers, checked on 14 November 2026 against the operator’s documentation
Name in the robots.txtOperatorPurposeSpecial feature
GooglebotGoogleGoogle Search, also AI OverviewsWithout a group of its own, “*” applies.
Googlebot-ImageGoogleGoogle ImagesWithout a group of its own, it follows the Googlebot group.
bingbotMicrosoftBing and CopilotWithout a group of its own, it follows an msnbot group, and only then “*”. Respects Crawl-delay.
ApplebotAppleSiri, Spotlight, SafariWithout a group of its own, it follows the Googlebot group.
DuckDuckBotDuckDuckGoDuckDuckGo’s own indexWithout a group of its own, “*” applies.
YandexBotYandexYandex SearchWithout a group of its own, it follows the Yandex group. Crawl-delay has had no effect since February 2018.
Storebot-GoogleGoogleGoogle ShoppingGoogle documents no fallback group; the tester calculates with “*” according to RFC 9309.
AdsBot-GoogleGoogleChecks landing pages of Google AdsSkips the “*” group. It can only be blocked with a group of its own.
Mediapartners-GoogleGoogleAdSense and Ad ManagerSkips the “*” group. It can only be blocked with a group of its own.

In addition there are 24 identifiers of AI providers, among them GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User. Since 2023, websites have increasingly blocked AI providers, in very different ways, and terms of use and robots.txt often contradict each other Longpre et al. 2024. A measurement of about 4 million domains shows that robots.txt remains the most important tool for this, and AI agents are mostly restricted there with Disallow Hoffmann et al. 2026. What each AI crawler is used for is explained in the AI crawler check, which also fetches your real robots.txt.

Write a robots.txt for common cases

People who need a robots.txt rarely want to learn technical terms. In a survey of 203 artists, the wish for the file was strong, but knowledge about it and access to their own website were often lacking Liu et al. 2025. The generator therefore asks only what crawlers should be allowed to read, and writes plain Allow and Disallow lines without tricks. With “Check in the tester”, the text lands in the tester above.

Generator · robots.txt for common cases

The robots.txt only controls crawling. It protects no data, because anyone can read it, and it removes no page from the Google index: a blocked URL can appear in the results without a description if other pages link to it.

What should crawlers be allowed to read?
Which areas to block?

With a slash at the end, the rule blocks only the directory. Without it, /intern also matches /internes-handbuch.

Two lines per name, for ?name= and &name=. That way “sort” does not also match “resort”.

With https:// and domain, otherwise the line is left out.

# robots.txt, erstellt mit dem robots.txt-Tester von aureum-tech.com (Stand 14. November 2026)
# Regelt nur das Crawlen. Schützt keine Daten und nimmt keine Seite aus dem Google-Index.
User-agent: *
Disallow:

The generator runs in your browser and sends nothing. The file belongs in the root directory as robots.txt, with a separate file for each subdomain.

The Sitemap line is worthwhile almost every time. Following links and sitemaps complement each other: across billions of URLs at Google, pages without incoming links and new content came into view faster through the sitemap Schonfeld and Shivakumar 2009. The line needs the full URL and applies to all groups.

What the robots.txt does not do

  • It removes nothing from the index. According to Google, robots.txt is not a way to keep a page out of Google. A blocked URL that other pages link to can appear in the results without a description. To get it out, use noindex, and Google only sees that if it is allowed to read the page.
  • It protects nothing. The file is public and lists the blocked paths, and the RFC says expressly that it is no substitute for access control. Whether a crawler abides by it is a matter of judgement for its operator Thelwall and Stuart 2006. Confidential material belongs behind a password.
  • It is a request. Even the first large surveys described the file as a rule that well-behaved crawlers follow, not as enforcement Sun et al. 2007a. In experiments with 130 named bots over 40 days, bots followed stricter rules less often, and AI search crawlers often did not fetch the file at all Kim et al. 2025.
  • It only works on names. Bots cannot be recognised reliably; none of the usual methods does so dependably in general Doran and Gokhale 2011. Anyone who arrives under a false name is not covered by any rule.
  • Hard blocks sit at the server. Against AI crawlers, blocks at the reverse proxy protected more effectively than robots.txt Liu et al. 2025. Conversely, some hosts and website builders turn crawlers away before the robots.txt even counts: in one Common Crawl extract this was about 1.68 per cent of websites Ansar et al. 2025. Then the best Allow line is no use.
  • The tester calculates with your text. Whether your server delivers exactly this file, with status 200 and as text, is checked by the AI crawler check with a real fetch.

The tester answers a preliminary question. Whether your pages really are in Google, and what is keeping them out if they are not, is shown by the SpotLens quick check. First steps for looking into it yourself are in the guide Website not found on Google, and how we make websites in Rodewisch and the Vogtland easier to find is under SEO. More tools for findability are under SEO and visibility.

Using this tool from an AI assistant

Assistants such as Claude or ChatGPT that speak the Model Context Protocol (MCP) can call this page’s calculation directly instead of rebuilding it. The server is public, needs no sign-in and stores nothing. The answer always includes the link to this page, the date of the data and the limits of the result.

Endpoint
https://aureum-tech.com/mcp
Tool
werkzeug_robots_txt_tester
Transport
Streamable HTTP, read-only
Inputs
NameRequiredMeaning
robots_txtyesThe complete text of the robots.txt, as it appears at /robots.txt.
pfadenoUp to 50 paths or full URLs, e.g. “/kontakt” or “https://www.firma.de/blog?seite=2”. For URLs, path and query count. Default “/”.
crawlernoProduct tokens of the crawlers, e.g. Googlebot, bingbot, GPTBot; “*” for every crawler without a group of its own. Known: Googlebot, Googlebot-Image, bingbot, Applebot, DuckDuckBot, YandexBot, Storebot-Google, AdsBot-Google, Mediapartners-Google, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Google-Agent, PerplexityBot, Perplexity-User, Applebot-Extended, meta-externalagent, meta-webindexer, meta-externalfetcher, Amazonbot, Amzn-SearchBot, Amzn-User, MistralAI-Training, MistralAI-Index, MistralAI-User, DuckAssistBot, YouBot, CCBot, Bytespider, *. Other names are checked according to RFC 9309. Default: Googlebot, bingbot, GPTBot, ClaudeBot, *.

Questions where an assistant should call it

  • “Can Googlebot fetch /wp-admin/admin-ajax.php with this robots.txt?”
  • “Why does my block under “User-agent: *” not apply to GPTBot?”
  • “Which line of my robots.txt blocks https://www.firma.de/shop?sort=preis for Bingbot?”
  • “Is “Disalow: /intern/” in my robots.txt a mistake, and does Google read the line anyway?”

How the connection works and what MCP can do in your own business is explained on the page about MCP integrations.

Common questions about robots.txt

Does “Disallow” remove a page from the Google index?

No. Disallow only forbids fetching. If other pages link to the address, Google can still include it in the results, without a description. To make a page disappear, Google must be allowed to read it and find a noindex there, in the meta tag or the X-Robots-Tag; otherwise the page must return 404 or 410 or sit behind a password. Blocking and noindex together do the opposite: Google never sees the noindex.

Why does my block under “User-agent: *” not apply to Googlebot?

Because there is a group of its own for Googlebot. A crawler follows the group with its name and only falls back to the catch-all group “*” if there is none. Whatever is blocked there must therefore also appear in every named group. For each result, the tester shows which group applies.

What wins when Allow and Disallow match the same path?

The longer rule, counted in characters. If they are equally long, Allow wins; the order in the file makes no difference. With “Disallow: /shop/” and “Allow: /shop/angebote/”, /shop/angebote/ therefore stays open and the rest of the shop does not.

How quickly does a changed robots.txt take effect at Google?

Google usually caches the file for up to 24 hours, and longer if refreshing fails. Until then the old rules apply. So check a change here first and only then upload it.

Can I protect confidential areas with robots.txt?

No. The file is public, anyone can open it, and it even reveals the blocked paths. The standard says so itself: it is no substitute for access control. Confidential material belongs behind a login or password protection on the server.

Does Google respect Crawl-delay or Noindex in robots.txt?

No. Google only reads user-agent, allow, disallow and sitemap. Noindex, nofollow and crawl-delay in robots.txt have had no effect at Google since 1 September 2019. Bing respects Crawl-delay, Yandex has not since February 2018, Applebot does not. The tester flags such lines.

Where does the robots.txt tester stop?

It calculates with the text you paste in, according to RFC 9309 and the documented quirks of the crawlers. Whether your server delivers exactly this file is checked by the AI crawler check with a real fetch. Whether your pages are in the index, and what is wrong if they are not, is shown by the SpotLens quick check. We fix whatever keeps pages out of Google on request.

What the tool is based on

The guideline values and notes above rest on these 13 scientific papers. Each was checked via its DOI or the publisher’s page (most recently on 14 November 2026): it exists, it has not been retracted, and its finding still holds for the question today.

  1. Sun, Y.; Zhuang, Z.; Giles, C. L. (2007): A large-scale study of robots.txt. Proceedings of the 16th International Conference on World Wide Web (WWW 2007), pp. 1123–1124

    Finding Five successive surveys of 7,593 websites show that robots.txt is used more and more. The file enforces nothing; it is well-behaved crawlers that abide by it.

    In the tool It underlies the warning above every result: robots.txt is a request, not a lock.

  2. Sun, Y.; Zhuang, Z.; Councill, I. G.; Giles, C. L. (2007): Determining Bias to Search Engines from Robots.txt. IEEE/WIC/ACM International Conference on Web Intelligence (WI'07), pp. 149–155

    Finding In 2,925 robots.txt files, the crawlers of large search engines were mostly favoured through groups of their own, and the preference was strongly linked to market share.

    In the tool This is why the tester compares several crawlers side by side and shows which group applies to which crawler: differences arise through named groups.

  3. Kolay, S.; D'Alberto, P.; Dasdan, A.; Bhattacharjee, A. (2008): A larger scale study of robots.txt. Proceedings of the 17th International Conference on World Wide Web (WWW 2008), pp. 1171–1172

    Finding A survey of about 6 million websites found, unlike the smaller 2007 study, no notable preference for a particular search engine.

    In the tool It explains in the text that findings on preference vary by sample. The tester therefore does not judge; it shows the group that applies for each crawler.

  4. Giles, C. L.; Sun, Y.; Councill, I. G. (2010): Measuring the web crawler ethics. Proceedings of the 19th International Conference on World Wide Web (WWW 2010), pp. 1101–1102

    Finding Most commercial crawlers behaved in line with the rules, but many repeatedly violated or misunderstood certain robots.txt rules.

    In the tool This is why the tester reports lines that crawlers read differently, and the generator writes only plain Allow and Disallow lines.

  5. Sun, Y.; Councill, I. G.; Giles, C. L. (2010): The Ethicality of Web Crawlers. 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), pp. 668–675

    Finding The behaviour of many crawlers was measured with decoy pages, each carrying a different robots.txt rule. How well they complied differed greatly; many commercial crawlers regularly violated certain rules.

    In the tool It supports the limit “a rule is not behaviour”: whether a crawler really complies is shown only by the server log, not by the tester.

  6. Thelwall, M.; Stuart, D. (2006): Web crawling ethics revisited: Cost, privacy, and denial of service. Journal of the American Society for Information Science and Technology 57 (13), pp. 1771–1779

    Finding The paper classifies the consequences of crawling for website operators by cost, privacy and overload. Whether a crawl is justifiable requires the crawler’s operator to make a judgement in each case.

    In the tool It explains why the tester points to password protection rather than a Disallow line for confidential areas.

  7. Schonfeld, U.; Shivakumar, N. (2009): Sitemaps: above and beyond the crawl of duty. Proceedings of the 18th International Conference on World Wide Web (WWW 2009), pp. 991–1000

    Finding A comparison of link following and sitemaps across billions of URLs at Google shows that the two complement each other well: sitemaps bring pages without incoming links and fresh content into view faster.

    In the tool This is why the tester lists the Sitemap lines separately, checks whether they are given as full URLs, and the generator writes one as well.

  8. Doran, D.; Gokhale, S. S. (2011): Web robot detection techniques: overview and limitations. Data Mining and Knowledge Discovery 22 (1–2), pp. 183–210

    Finding A comparison of the usual methods for detecting bots shows that none works reliably in general. Some bots behave badly or put considerable load on the server.

    In the tool It underlies the limit that bots cannot be recognised reliably: a rule only works on crawlers that give their name and abide by it.

  9. Kim, T.; Bock, K.; Luo, C. et al. (2025): Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. Proceedings of the 2025 ACM Internet Measurement Conference (IMC 2025), pp. 541–557

    Finding In controlled experiments with 130 named bots over 40 days, bots followed stricter rules less often, and AI search crawlers often did not fetch the robots.txt at all.

    In the tool This is why every “blocked” carries the note that the block is a request, and “Block everything” in the generator carries the recommendation to set up real protection at the server.

  10. Longpre, S.; Mahari, R.; Lee, A. et al. (2024): Consent in Crisis: The Rapid Decline of the AI Data Commons. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp. 108042–108087

    Finding An audit of 14,000 domains behind AI training data found, within a year, a sharp rise in blocks, large differences between AI providers and frequent contradictions between terms of use and robots.txt.

    In the tool This is why the AI crawlers can be chosen individually, and for the question “training yes or no” the tester refers to the AI crawler check.

  11. Liu, E.; Luo, E.; Shan, S. et al. (2025): Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. Proceedings of the 2025 ACM Internet Measurement Conference (IMC 2025), pp. 78–99

    Finding Measurements and a survey of 203 artists show a great need for robots.txt, but obstacles in technical knowledge and in access to their own website. Blocks at the reverse proxy protected more effectively.

    In the tool This is why there is a generator for common cases without technical terms, and advice to involve the host for a hard block.

  12. Hoffmann, F.; Goergens, T.; Khosla, S.; Bajpai, V. (2026): From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review 56 (2), pp. 29–41

    Finding A measurement of about 4 million domains shows that robots.txt remains the most important tool for allowing or forbidding crawler access. AI agents are restricted there mainly with Disallow lines.

    In the tool It explains why the tester sticks to robots.txt and checks AI identifiers on an equal footing with search engines.

  13. Ansar, M.; Sperotto, A.; Holz, R. (2025): Web Crawl Refusals: Insights From Common Crawl. Passive and Active Measurement (PAM 2025), Lecture Notes in Computer Science, pp. 197–214

    Finding About 1.68 per cent of websites in a Common Crawl snapshot expressly turned the crawler away, often through large hosts and site-builder providers, ranging from a block to a challenge page.

    In the tool This establishes the limit that a permissive robots.txt is no use if the host or bot protection turns the crawler away first.

Standards, authorities and manufacturer data

Mon–Fri 8:00–17:00

Send us the copied result. We check the robots.txt and sitemap of your website and fix whatever keeps pages out of Google, in Rodewisch and throughout the Vogtland.

+49 3744 4399760[email protected]

Mon–Fri 8:00–17:00 · Straße der Jugend 5, 08228 Rodewisch

Have your robots.txt checked
+49 3744 4399760Mon–Fri 8:00–17:00
CallCallback

PARTS LIST · SHEET 1/1

What this page is made of

Values measured at build time. With JavaScript, your browser measures them again live.

ITEMPARTTYPESIZE
01index.htmlDocument≈ 29 kB
02styles.cssStylesheet≈ 20 kB
03scripts.jsScript≈ 13 kB
04archivo-variable.woff2Font≈ 58 kB
05ibm-plex-mono.woff2Font≈ 30 kB
06favicon.svgImage< 1 kB

Advertising or analytics cookies: none. Without your consent, the page loads nothing from third-party servers apart from Cloudflare, our delivery service. Fonts, scripts and images are hosted by us. That is how we build your IT, too.

OUTSIDE OFFICE HOURS

Feel free to call anyway

Our regular office hours are Monday to Friday, 8:00–17:00.

In the evening and at weekends the number is forwarded to us, and often someone will pick up. We cannot promise that, though.

If nobody answers, leave a short message on the voicemail: your name, your number and what it is about. We will call back as soon as we can, on the next working day at the latest.

The call costs nothing. If someone has to start work straight away outside office hours, the surcharge in the price list applies.

Call now+49 3744 4399760

Prefer to write? Use the contact form or e-mail [email protected]. We will read it on the next working day.

DIAGNOSIS · AT-K2

What do you need help with?

↵ SELECTESC CLOSECALL: +49 3744 4399760

Custom settings

Which services may load?

Strictly necessary

The choice you make here and your settings for colour scheme and motion. These are kept as small entries on your device and are never sent to a server. They are needed for the site to work and therefore cannot be switched off.