Tool AT-W29REV. 2026-A

AI Crawler Check: Can ChatGPT, Claude and Perplexity read your website?

Whether an AI assistant may read your website is set in the robots.txt: each crawler has its own group, otherwise the catch-all rule “User-agent: *” applies. What matters is the purpose. GPTBot and ClaudeBot collect for training, OAI-SearchBot and Claude-SearchBot for AI search, and ChatGPT-User fetches a page when someone asks in the chat. The check reads your rules for 27 crawlers, checks the llms.txt and writes the matching block for you.

  • free
  • no sign-up
  • 5-minute cache

Enter a domain, read the access per crawler

Fetch · robots.txt, llms.txt and home page

The check reads the rules as they stand on the server today and evaluates them according to RFC 9309. Whether a crawler complies is up to its operator.

For example your-company.de. If the home page redirects to www., the check looks there.

Without an entry, the home page. With /blog or /services you can see whether exactly that area is open.

The domain is sent to our server, which retrieves robots.txt, llms.txt and the start of the home page. The domain and result are not logged. For follow-up checks, both are held in RAM for up to five minutes.

Without a fetch: paste the robots.txt text

For example for a file that is not online yet. The text stays in the browser; the path from above applies.

Result

Enter a domain above. The fetch takes a few seconds.

Which AI crawlers exist and why they read

The big providers do not send one crawler but several, depending on purpose. That is the most important point when setting things up: anyone who shuts out “the AI” usually also drops out of AI search and is no longer named as a source there. Since 2023 websites have increasingly blocked AI providers, and in very different ways; researchers often found contradictions between terms of use and robots.txt Longpre et al. 2024. That is why the check shows each crawler separately.

TRAINING, E.G. GPTBOT, CLAUDEBOT, CCBOTYOUR WEBSITEROBOTS.TXTCRAWLER COLLECTSFUTUREMODELAI SEARCH, E.G. OAI-SEARCHBOT, PERPLEXITYBOTYOUR WEBSITEROBOTS.TXTSEARCH INDEXANSWER WITHSOURCEUSER FETCH, E.G. CHATGPT-USERQUESTION IN CHATFETCH OF ONE PAGEYOUR WEBSITETRAINING, E.G. GPTBOT,CLAUDEBOT, CCBOTYOUR WEBSITEROBOTS.TXTCRAWLER COLLECTSFUTURE MODELAI SEARCH, E.G. OAI-SEARCHBOT,PERPLEXITYBOTYOUR WEBSITEROBOTS.TXTSEARCH INDEXANSWER WITH SOURCEUSER FETCH, E.G. CHATGPT-USERQUESTION IN CHATFETCH OF ONE PAGEYOUR WEBSITE
FIG. 1 · THREE WAYS AI READS YOUR WEBSITEGOLD = YOUR RULE IN THE ROBOTS.TXT · DASHED = RULE DOES NOT ALWAYS APPLY
Four types, as stated by the operators
TypeEntriesWhat they do
AI search11Builds the index from which an AI assistant or a search engine gets answers with sources. If blocked, the page appears there less often as a source, or not at all.
Training6Collects content that language models learn from. A ban affects future training runs, not models that are already trained.
Training switch2Not a crawler of its own. The rule tells the operator whether it may use what it fetches anyway for training.
User fetch7Fetches a single page because a person has just asked about it. Several operators state that robots.txt does not apply to this, or not always.

Two entries deserve a second look. Common Crawl does not train a model itself, but it builds the largest freely available archive of the web, and that is one of the most important sources for the pre-training of large language models Baack 2024. The well-known training corpus C4, for instance, was built from a single such snapshot Dodge et al. 2021. Anyone who refuses training therefore blocks CCBot too; but no rule takes back what is already in a snapshot. The second is Googlebot: it is on the list because the AI Overviews and AI Mode of Google Search build on its index. Nobody who wants to be found on Google blocks it.

All 27 crawlers covered

Crawler list, checked on 14 November 2026 against each operator’s documentation
Name in the robots.txtOperatorPurposerobots.txt appliesWhat the operator says
OAI-SearchBotOpenAIAI searchyes, according to the operatorFeeds the search in ChatGPT. If blocked, the site does not appear there as a source in answers.
Claude-SearchBotAnthropicAI searchyes, according to the operatorFeeds the search in Claude. If blocked, the page appears less often in results there.
GooglebotGoogleAI searchyes, according to the operatorThe crawler of Google Search. AI Overviews and AI Mode also draw on this index; they are controlled there through nosnippet, max-snippet and noindex.
PerplexityBotPerplexityAI searchyes, according to the operatorBuilds the index from which Perplexity links sources. According to the operator, it is not used to train models.
ApplebotAppleAI searchyes, according to the operatorCrawler for search in Siri, Spotlight and Safari. According to Apple, the data may also go into the training of Apple’s models; Applebot-Extended exists for that.
meta-webindexerMetaAI searchyes, according to the operatorIndexes for search in Meta AI so that Meta can cite and link content.
Amzn-SearchBotAmazonAI searchyes, according to the operatorFor search functions in Amazon products such as Alexa; according to Amazon, not for training.
bingbotMicrosoftAI searchyes, according to the operatorThe crawler of Bing. Microsoft names no separate AI identifier; from February 2027 the settings for training and AI answers are addressed to Bingbot.
MistralAI-IndexMistralAI searchyes, according to the operatorCrawls only for the search index; according to Mistral, not for training.
DuckAssistBotDuckDuckGoAI searchyes, according to the operatorFetches pages in real time for DuckDuckGo’s AI answers, with source attribution. According to the operator, not for training.
YouBotYou.comAI searchyes, according to the operatorCrawler of the You.com search engine. Follows robots.txt including Crawl-delay.
GPTBotOpenAITrainingyes, according to the operatorCollects content that may go into the training of OpenAI’s models. According to OpenAI, blocked means: do not use for training.
ClaudeBotAnthropicTrainingyes, according to the operatorCollects content that may go into the training of Anthropic’s models. Anthropic follows robots.txt and Crawl-delay too.
meta-externalagentMetaTrainingyes, according to the operatorCollects for training AI models and for Meta’s products.
AmazonbotAmazonTrainingyes, according to the operatorImproves Amazon’s products, which can include training Amazon’s AI models. It does not honour Crawl-delay.
MistralAI-TrainingMistralTrainingyes, according to the operatorCollects content for training data of the Mistral models.
CCBotCommon CrawlTrainingyes, according to the operatorBuilds a freely accessible archive of the web. Common Crawl trains nothing itself, but the archive is one of the most important sources of training data.
Google-ExtendedGoogleTraining switchyes, according to the operatorNot a crawler of its own but a switch: may Google’s fetches go into the training of future Gemini models and into the grounding of Gemini answers? According to Google, it has no effect on Google Search.
Applebot-ExtendedAppleTraining switchyes, according to the operatorNot a crawler of its own. If blocked, Apple does not use the content for training its models; the page stays in search.
ChatGPT-UserOpenAIUser fetchnot alwaysFetches a page when someone asks about it in ChatGPT. According to OpenAI, robots.txt rules may not apply to this.
Claude-UserAnthropicUser fetchyes, according to the operatorRetrieves pages when users ask a question in Claude. Anthropic says its bots follow robots.txt.
Google-AgentGoogleUser fetchnot alwaysBrowses on behalf of a user and carries out actions. According to Google, user fetches usually bypass robots.txt.
Perplexity-UserPerplexityUser fetchnot alwaysFetches pages when a user asks. According to Perplexity, it usually ignores robots.txt.
meta-externalfetcherMetaUser fetchnot alwaysFetches individual links at the user’s request, also for AI agents. According to Meta, it can bypass robots.txt.
Amzn-UserAmazonUser fetchnot alwaysFetches current data for a user’s question, for example for Alexa. According to Amazon, it does not follow all robots.txt rules.
MistralAI-UserMistralUser fetchyes, according to the operatorFetches a page when a user asks and links it as a source. According to Mistral, the robots.txt group determines which pages qualify.
BytespiderByteDanceunclearnot documentedNo operator documentation found. Purpose and handling of robots.txt are not documented.

Not included: names from bot directories that the operator itself does not give (such as anthropic-ai or cohere-ai). According to its own documentation, Cohere currently runs no crawler; Microsoft names no AI identifier of its own, its settings for AI are addressed to bingbot. As of 14 November 2026.

Block AI training, allow AI search: the matching block

Most businesses want the same thing: to appear as a source in the answers of ChatGPT, Claude or Perplexity, but not to supply training material unasked. The generator is preset for that. It writes each group separately and only with Allow and Disallow, without wildcards, because even commercial crawlers repeatedly misread certain rules Giles et al. 2010. You paste the block into your existing robots.txt; the group “User-agent: *” stays as it is.

Generator · robots.txt block

May AI providers use your content for training?

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training, CCBot, Bytespider

May AI search services read your pages and name them as a source?

OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot, YouBot

May assistants fetch a page when someone asks for it?

ChatGPT-User, Claude-User, Google-Agent, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User

Whatever is blocked in your robots.txt under “User-agent: *” belongs here. Named crawlers do not inherit it.

# KI-Crawler, erstellt mit dem KI-Crawler-Check von aureum-tech.com (Crawler-Liste Stand 14. November 2026)
# Training: gesperrt · KI-Suche: erlaubt · Nutzerabrufe: erlaubt
# Jeder Crawler befolgt nur die Gruppe mit seinem Namen. Was für alle gesperrt sein soll, steht deshalb in jeder Gruppe.

# Training (Schalter wie Google-Extended crawlen nicht selbst, sie regeln nur die Nutzung)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# KI-Suche: liefert Antworten mit Quellenangabe
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: MistralAI-Index
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: YouBot
Allow: /

# Nutzerabrufe: laut mehreren Betreibern nicht immer an robots.txt gebunden
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Google-Agent
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: meta-externalfetcher
Allow: /

User-agent: Amzn-User
Allow: /

User-agent: MistralAI-User
Allow: /

# Gemeinsames Trainingssignal (IETF-Entwurf vom 30.10.2026, mehrere Anbieter haben es angekündigt)
User-agent: __usage_ai-training
Disallow: /

The generator runs in the browser and sends nothing.

Generator · llms.txt outline

Only pages that your robots.txt releases for AI search. The outline follows the proposal on llmstxt.org.

# Name Ihrer Firma

> Ein Satz: Wer Sie sind, was Sie anbieten und wo.

## Wichtige Seiten

- [Leistungen](https://ihre-firma.de/leistungen): Was Sie anbieten
- [Kontakt](https://ihre-firma.de/kontakt): Telefon, Adresse, Öffnungszeiten

## Optional

- [Impressum](https://ihre-firma.de/impressum)

Runs in the browser. The file goes into the root directory of the website as llms.txt.

The most common trap: groups inherit nothing

A crawler follows exactly the group with its name. If it exists, the catch-all rule no longer applies to it, not even partly. In the example on the right, “User-agent: *” blocks the area /intern/, but GPTBot gets its own group with “Allow: /”.

The result for /intern/preise: ClaudeBot is blocked (line 2: Disallow: /intern/ (group *)), whereas GPTBot may read (line 5: Allow: / (group GPTBot)). Anything that is to be blocked for everyone therefore belongs in every named group. In the generator you enter such areas once and it repeats them everywhere.

User-agent: *
Disallow: /intern/

User-agent: GPTBot
Allow: /

The shared training signal

Since 30 October 2026 there has been an IETF draft meant to shorten the blocking of crawlers one by one: a group “User-agent: __usage_ai-training” with “Disallow: /” tells all participating providers at once that you do not want training. Paths work as usual; “Allow: /presse/” exempts the press kit. The group is not a crawling instruction, and the catch-all rule “User-agent: *” expressly does not count for it. Because the draft is young and not every provider reads it yet, the generator writes both: the individual groups and the signal.

Who honours the training signal, by their own account
ProviderFrom when
Anthropicsince 30 October 2026; a named ClaudeBot group takes precedence
Microsoft (Bing)from 1 February 2027
Googleplanned for 2027; without a group of its own for Google-Extended, Google will then treat a No like a blocked Google-Extended
OpenAI“in the coming months”; the GPTBot rules remain

Source: draft-kelley-aipref-training-preference-signal-00, without standard status, expires on 3 May 2027. As of 14 November 2026.

How a crawler reads your robots.txt

Since September 2022 the procedure has been laid down as RFC 9309, and the check calculates by it too. The file always sits at /robots.txt in the root directory and applies only to exactly that host: if your-company.de redirects to www.your-company.de, the file on www. counts. The crawler looks for the group with its name, upper or lower case does not matter; if there is none, it takes “User-agent: *”, and if there is none of that either, it may read everything. Within the group the longest matching rule wins; if equal, Allow. An asterisk stands for any characters, a dollar sign at the end for the end of the address.

As early as 2007 a survey of 7,593 websites showed that the file is used more and more, and also what it is: a rule that decent crawlers follow, not enforcement Sun et al. 2007. Nothing has changed there.

What the status code of the robots.txt means
ResponseRFC 9309Google
200, text fileRules apply, at least 500 KiB are readlikewise, everything after 500 KiB is ignored
200, but a web pageno valid line, so no rulelikewise
404 and other 4xxno robots.txt, everything allowedeverything allowed, except for 429
429 too many requestslike 4xxlike a server error
5xx or unreachableeverything blockedno crawling for 12 hours, then the last good version for up to 30 days
Redirectfollow at least fiveat least five, then like 404

Google usually caches the file for up to 24 hours, Amazon for up to 30 days. Changes therefore do not take effect immediately. Our fetch follows at most three redirects. As of 14 November 2026.

Directives on the pages themselves

Besides the robots.txt there are directives per page, as a meta tag in the head or as an X-Robots-Tag in the server response. The check reads them on the home page. For AI these matter most: noindex takes the page out of the search index and so also out of AI Overviews; nosnippet forbids the text excerpt at Google, including in AI Overviews and AI Mode. Amazon reads noarchive as “not for training”. From 1 February 2027 Microsoft introduces its own directives for training and grounding (ai-training, ai-grounding) and from March 2027 no longer reads noarchive and nocache as an AI directive. Directives such as “noai” are widespread, but not documented by any operator on our list.

What the robots.txt does not do

  • It is a request. In trials with 130 named bots over 40 days, bots followed strict rules less often, and AI search crawlers often did not even request the file Kim et al. 2025.
  • It only works on names. With specially prepared websites it was possible to show which fetcher feeds which AI system, including several that their operator does not disclose Seiden et al. 2026. Anyone who arrives under a different name is not covered by any rule.
  • User fetches partly bypass it. OpenAI, Perplexity, Meta, Amazon and Google themselves state that their fetches at a user’s request do not always follow robots.txt. Anthropic and Mistral name the file as the control for these as well.
  • Hard blocks sit on the server. Artists often lacked the knowledge or the access to their own website for robots.txt; blocks at the reverse proxy protected more effectively Liu et al. 2025. Conversely, some hosts and site builders block crawlers without the owner knowing: in one Common Crawl snapshot about 1.68 per cent of websites expressly turned the crawler away Ansar et al. 2025. If your home page answers our fetch with a challenge page, the check reports it.
  • Our fetch is not an AI crawler. The server fetches under its own identifier. If a firewall blocks crawlers by name, for instance through a Cloudflare setting, the check does not see it.

And legally?

Under § 44b para. 3 UrhG (German Copyright Act), text and data mining of freely accessible works is permitted unless the rights holder has reserved that use, and for content online only a reservation in machine-readable form counts. The robots.txt is machine-readable, but whether it suffices as a reservation has not been conclusively settled. A legal study argues for common-law countries that ignoring the file can, in some circumstances, give rise to liability Chang and He 2025; for Germany this has to be assessed separately. Anyone who wants to declare a reservation should therefore also put it in the legal notice (Impressum) or in the terms of use. This is not legal advice.

llms.txt: what the file can do and what it cannot

The llms.txt was proposed by Jeremy Howard on 3 September 2024; the description is on llmstxt.org. It is a Markdown file at /llms.txt: a heading with the name, which is the only mandatory part, then a short description as a quotation, optionally a few sentences, and sections with lists of links to the most important pages. A section “Optional” holds what an assistant can leave out.

A measurement of about 4 million domains found the file mainly at tech companies, and some of their links pointed to pages that their own robots.txt blocks Hoffmann et al. 2026. That is exactly what the check tests. A signpost to blocked pages leads nowhere.

To be honest: in its documentation on AI Overviews Google writes that no new machine-readable files are needed for them, and none of the operators covered here says in its crawler documentation that it reads the llms.txt of other websites. So it is not a lever for visibility. It still makes sense as a short, clean self-description that an assistant can read during a user fetch, and because it is done in ten minutes.

Our own is at aureum-tech.com/llms.txt. The generator above builds the outline for yours.

Being allowed to read is not yet being named

The check answers a preliminary question. Whether ChatGPT names your company when someone asks for an electrician in Plauen depends on the content of your pages and on what others write about you. In trials, the visibility of sources in AI answers could be raised by up to 40 per cent through how the content was prepared, with different methods depending on the field Aggarwal et al. 2024. At Google, a measurement of 55,393 search queries showed an AI Overview in 13.7 per cent of cases, and for questions in 64.7 per cent, and almost 30 per cent of the domains cited there were not among the hits on the first page Xu et al. 2026.

Named does not mean correctly cited either. In four AI search engines, on average only a little over half of the sentences were backed by the sources given Liu et al. 2023. And anyone who blocks Google’s AI crawler is used less often as a source in AI Overviews, although Google writes that the switch does not affect search Grossman et al. 2026.

The SpotLens quick check shows how visible your company really is in ChatGPT, Perplexity and Google’s AI. How we build content and structured data so that AI searches cite them is covered under AI-SEO. If you would like to look for yourself first, the guide explains how to check your visibility in ChatGPT and what AI visibility for businesses means. You can test individual rules in the robots.txt tester; further tools for being found are under SEO and visibility.

What the research says

The assessment in the check rests on 15 peer-reviewed papers. The most important findings follow; the full list with review notes is further down.

  • Blocks are increasing, and unevenly. Within a year, large parts of the sources behind AI training data were blocked, differing by provider Longpre et al. 2024. That is why the check shows crawlers individually.
  • Rules are followed selectively. The stricter the rule, the less often bots followed it Kim et al. 2025. That is why the check measures nothing it cannot measure, and says so.
  • Training runs through archives. Common Crawl is one of the most important sources for pre-training Baack 2024, and corpora such as C4 are built from single snapshots Dodge et al. 2021. That is why the check warns if CCBot stays open.
  • AI Overviews choose their own sources. Many cited domains are not on page one Xu et al. 2026, and blocked AI crawlers go along with fewer mentions Grossman et al. 2026.
  • llms.txt is young and often contradictory. Links in the file sometimes point to blocked pages Hoffmann et al. 2026; the check tests each link in the file.

Using this tool from an AI assistant

Assistants such as Claude or ChatGPT that speak the Model Context Protocol (MCP) can call this page’s calculation directly instead of rebuilding it. The server is public, needs no sign-in and stores nothing. The answer always includes the link to this page, the date of the data and the limits of the result.

Endpoint
https://aureum-tech.com/mcp
Tool
werkzeug_ki_crawler_check
Transport
Streamable HTTP, read-only
Inputs
NameRequiredMeaning
domainnoDomain of the website, e.g. your-company.de or https://www.your-company.de. A path is ignored, robots.txt always sits in the root directory. Public websites only, port 80 or 443.
robots_txtnoInstead of the domain: the text of a robots.txt, which is evaluated without a fetch. Do not give it together with domain.
pfadnoPath to check, e.g. /blog/article. Without it, the home page “/”.

Questions where an assistant should call it

  • “May ChatGPT read my website handwerk-muster.de?”
  • “Does our robots.txt block the AI search of Perplexity and ChatGPT by mistake?”
  • “How do I forbid AI training but stay visible in the search of ChatGPT and Claude?”
  • “Does www.beispiel-praxis.de have an llms.txt, and do its links point to blocked pages?”

How the connection works and what MCP can do in your own business is explained on the page about MCP integrations.

Common questions about AI crawlers, robots.txt and llms.txt

How do I block AI training without dropping out of ChatGPT search?

OpenAI keeps the two apart: GPTBot collects for training, OAI-SearchBot for search in ChatGPT, and each setting works on its own. In the robots.txt, GPTBot therefore gets “Disallow: /” and OAI-SearchBot gets “Allow: /”. The same applies to ClaudeBot and Claude-SearchBot at Anthropic; at Google you block Google-Extended and leave Googlebot alone. The generator on this page writes the whole block for all providers covered.

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects content that may go into the training of future models. OAI-SearchBot builds the index from which ChatGPT names sources in a search; according to OpenAI, a site that blocks it does not appear in answers there. ChatGPT-User fetches a single page because someone in the chat has just asked about it. According to OpenAI, robots.txt rules may not apply to this kind of fetch.

Will my website drop out of Google if I block Google-Extended?

No. Google-Extended is not a crawler of its own but a switch for whether Google may use what it has fetched for Gemini training and for grounding Gemini answers. According to Google it affects neither inclusion in search nor ranking. The AI Overviews in Google Search depend on Googlebot and on snippet rules such as nosnippet. A 2026 measurement did find, however, that pages blocking Google’s AI crawler were cited less often as a source in AI Overviews.

Do I need an llms.txt?

It is not mandatory. The llms.txt is a proposal by Jeremy Howard from September 2024: a Markdown file with a heading, a short description and links to the most important pages. Google states explicitly that AI Overviews need no new machine-readable files, and none of the operators covered here says in its crawler documentation that it reads it. It does no harm and is quickly done; the generator provides the outline. More important is that its links do not point to blocked pages.

Does every crawler follow the robots.txt?

No. The robots.txt is a request, not a lock. Measurements show that bots follow stricter rules less often and that AI search crawlers often do not request the file at all. According to their operators, user fetches such as ChatGPT-User, Perplexity-User or meta-externalfetcher partly bypass the robots.txt, and anyone who collects without giving a name appears in no rule. If you really want to block, you need a block on the server or at your host.

Does a block also apply to models that are already trained?

No. OpenAI and Anthropic describe the block as a signal for future training. Training data comes from fixed snapshots of the web, such as the Common Crawl archive; what is already there stays there. That is why it is worth including CCBot if you want to refuse training.

What does “User-agent: __usage_ai-training” mean?

It is the shared training signal from an IETF draft of 30 October 2026. A group with this name and “Disallow: /” tells all participating providers at once that you do not want training. Anthropic says it already honours it; Microsoft will do so from 1 February 2027, Google plans to honour it in 2027, and OpenAI in the coming months. It is not a crawling instruction, and the group “User-agent: *” expressly does not count for it.

Where does the AI Crawler Check stop?

It reads the rules of your robots.txt, the llms.txt and the robots directives on the home page, and tells you per crawler what is allowed. It does not measure whether a crawler complies, nor whether an AI assistant knows and recommends your company. The SpotLens quick check shows that; how content has to be built so that AI searches cite it is covered under AI-SEO. We can set the files up for you on request.

What the tool is based on

The guide values and notes above are based on these 15 scientific papers. Each one has been checked via its DOI or the publisher’s page (most recently on 14 November 2026): it exists, it has not been retracted, and its findings still hold for this question today.

  1. Sun, Y.; Zhuang, Z.; Giles, C. L. (2007): A large-scale study of robots.txt. Proceedings of the 16th International Conference on World Wide Web (WWW 2007), pp. 1123–1124

    Finding Five surveys of 7,593 websites show that robots.txt is used more and more. The file is not enforcement but a rule that decent crawlers follow.

    In the tool Sets out the page’s principle: robots.txt is a request to crawlers, not a lock.

  2. Giles, C. L.; Sun, Y.; Councill, I. G. (2010): Measuring the web crawler ethics. Proceedings of the 19th International Conference on World Wide Web (WWW 2010), pp. 1101–1102

    Finding Most commercial crawlers behaved in line with the rules, but many repeatedly violated or misunderstood certain robots.txt rules.

    In the tool That is why the generator writes each group separately and only with Allow and Disallow, without wildcards.

  3. Kim, T.; Bock, K.; Luo, C. et al. (2025): Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. Proceedings of the 2025 ACM Internet Measurement Conference (IMC 2025), pp. 541–557

    Finding In controlled trials with 130 named bots over 40 days, bots followed stricter rules less often; AI search crawlers often did not request the robots.txt at all.

    In the tool Underpins the limit “read rules, do not measure behaviour” and the note on blocked user fetches.

  4. Liu, E.; Luo, E.; Shan, S. et al. (2025): Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. Proceedings of the 2025 ACM Internet Measurement Conference (IMC 2025), pp. 78–99

    Finding Measurements and a survey of 203 artists show great demand for robots.txt, but hurdles in knowledge and in access to their own website. Blocks at the reverse proxy protected more effectively but are not widespread.

    In the tool Hence the recommendation to involve the host or proxy for a hard block, and the generator, which presumes no technical vocabulary.

  5. Seiden, S.; Ren, T.; Zhang, C. et al. (2026): Identifying AI Web Scrapers Using Canary Tokens. Proceedings of the 2026 ACM SIGSAC Conference on Computer and Communications Security (CCS 2026), pp. 6998–7012

    Finding With specially prepared websites and 22 AI systems it was possible to determine which fetcher feeds which model, including several that their operator does not disclose.

    In the tool Supports the limitation that the table only knows crawlers that their operator documents under its own name.

  6. Longpre, S.; Mahari, R.; Lee, A. et al. (2024): Consent in Crisis: The Rapid Decline of the AI Data Commons. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp. 108042–108087

    Finding An audit of 14,000 domains behind AI training data found, within a year, a sharp rise in blocks, large differences between AI providers and frequent contradictions between terms of use and robots.txt.

    In the tool That is why the check shows each crawler individually instead of merely reporting “AI allowed” or “AI blocked”.

  7. Baack, S. (2024): A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024), pp. 2199–2208

    Finding Common Crawl is the largest freely available collection of web data and one of the most important sources for the pre-training of large language models, often with a large share of the training material.

    In the tool That is why CCBot is listed under training, and the check warns if individual training crawlers are blocked but CCBot is not.

  8. Dodge, J.; Sap, M.; Marasović, A. et al. (2021): Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pp. 1286–1305

    Finding The training corpus C4 was built from a single Common Crawl snapshot and contains text from unexpected sources such as patent documents and military sites.

    In the tool Shows that training corpora are built from fixed snapshots: what is once in there is not reached by a later ban.

  9. Aggarwal, P.; Murahari, V.; Rajpurohit, T. et al. (2024): GEO: Generative Engine Optimization. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2024), pp. 5–16

    Finding Content can be prepared so that AI search engines cite it up to 40 per cent more visibly; what works depends on the field.

    In the tool Draws the distinction: access is the precondition, the content decides whether it is cited. For that the page points to AI-SEO.

  10. Liu, N.; Zhang, T.; Liang, P. (2023): Evaluating Verifiability in Generative Search Engines. Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001–7025

    Finding In four AI search engines, on average only a little over half of the sentences were fully backed by sources, and about a quarter of the citations did not support the sentence.

    In the tool States the limitation that free access does not promise a correct citation: the check tests reading, not the answer.

  11. Xu, H.; Iqbal, U.; Montgomery, J. M. (2026): Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact. Proceedings of the 2026 ACM Internet Measurement Conference (IMC 2026), pp. 529–546

    Finding Across 55,393 search queries, an AI Overview appeared in 13.7 per cent of cases, and for questions in 64.7 per cent. Almost 30 per cent of the cited domains were not among the hits on the first page.

    In the tool Explains why Googlebot is in the check: AI Overviews choose sources from the index, but not only from the top hits.

  12. Grossman, R.; Liu, S.; Chen, M. K. et al. (2026): How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2026), pp. 448–459

    Finding Across 11,500 search queries, the sources of Google Search, AI Overviews and Gemini differed greatly. Websites that block Google’s AI crawler were used less often as a source in AI Overviews.

    In the tool The note on a blocked Google-Extended sets this finding beside Google’s statement that the switch does not affect search.

  13. Hoffmann, F.; Goergens, T.; Khosla, S.; Bajpai, V. (2026): From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review 56, pp. 29–41

    Finding A measurement of about 4 million domains shows: robots.txt remains the most important tool; AI-specific files such as ai.txt and llms.txt appear mainly at tech companies. Some links in llms.txt files point to pages that the robots.txt blocks.

    In the tool Hence the test of whether links in the llms.txt point to blocked pages.

  14. Chang, C.-Y.; He, X. (2025): The liabilities of robots.txt. Computer Law & Security Review 58, article 106176

    Finding With generative AI, ignoring robots.txt weighs more heavily. The paper argues for common-law countries that the file can, in some circumstances, work as a unilateral contract or as a warning with consequences for liability.

    In the tool Supports the note that robots.txt is more than technology, with the proviso that the legal position in Germany has to be assessed separately.

  15. Ansar, M.; Sperotto, A.; Holz, R. (2025): Web Crawl Refusals: Insights From Common Crawl. Passive and Active Measurement (PAM 2025), Lecture Notes in Computer Science, pp. 197–214

    Finding About 1.68 per cent of websites in a Common Crawl snapshot expressly turned the crawler away, often through large hosts and site-builder providers, ranging from a block to a challenge page.

    In the tool Hence the note when the home page answers with a challenge page: bot protection blocks independently of the robots.txt.

Standards, authorities and manufacturer data

Mon–Fri 8:00–17:00

Send us the copied result. We set up robots.txt, llms.txt and structured data so that AI searches read your pages and training only takes place where you want it, in Rodewisch and across the whole Vogtland.

+49 3744 4399760[email protected]

Mon–Fri 8:00–17:00 · Straße der Jugend 5, 08228 Rodewisch

Have AI access set up
+49 3744 4399760Mon–Fri 8:00–17:00
CallCallback

PARTS LIST · SHEET 1/1

What this page is made of

Values measured at build time. With JavaScript, your browser measures them again live.

ITEMPARTTYPESIZE
01index.htmlDocument≈ 29 kB
02styles.cssStylesheet≈ 20 kB
03scripts.jsScript≈ 13 kB
04archivo-variable.woff2Font≈ 58 kB
05ibm-plex-mono.woff2Font≈ 30 kB
06favicon.svgImage< 1 kB

Advertising or analytics cookies: none. Without your consent, the page loads nothing from third-party servers apart from Cloudflare, our delivery service. Fonts, scripts and images are hosted by us. That is how we build your IT, too.

OUTSIDE OFFICE HOURS

Feel free to call anyway

Our regular office hours are Monday to Friday, 8:00–17:00.

In the evening and at weekends the number is forwarded to us, and often someone will pick up. We cannot promise that, though.

If nobody answers, leave a short message on the voicemail: your name, your number and what it is about. We will call back as soon as we can, on the next working day at the latest.

The call costs nothing. If someone has to start work straight away outside office hours, the surcharge in the price list applies.

Call now+49 3744 4399760

Prefer to write? Use the contact form or e-mail [email protected]. We will read it on the next working day.

DIAGNOSIS · AT-K2

What do you need help with?

↵ SELECTESC CLOSECALL: +49 3744 4399760

Custom settings

Which services may load?

Strictly necessary

The choice you make here and your settings for colour scheme and motion. These are kept as small entries on your device and are never sent to a server. They are needed for the site to work and therefore cannot be switched off.