Tool AT-W29REV. 2026-A
AI Crawler Check: Can ChatGPT, Claude and Perplexity read your website?
Whether an AI assistant may read your website is set in the robots.txt: each crawler has its own group, otherwise the catch-all rule “User-agent: *” applies. What matters is the purpose. GPTBot and ClaudeBot collect for training, OAI-SearchBot and Claude-SearchBot for AI search, and ChatGPT-User fetches a page when someone asks in the chat. The check reads your rules for 27 crawlers, checks the llms.txt and writes the matching block for you.
- free
- no sign-up
- 5-minute cache
01 / FETCH
Enter a domain, read the access per crawler
Fetch · robots.txt, llms.txt and home page
The check reads the rules as they stand on the server today and evaluates them according to RFC 9309. Whether a crawler complies is up to its operator.
Without a fetch: paste the robots.txt text
For example for a file that is not online yet. The text stays in the browser; the path from above applies.
02 / CRAWLERS
Which AI crawlers exist and why they read
The big providers do not send one crawler but several, depending on purpose. That is the most important point when setting things up: anyone who shuts out “the AI” usually also drops out of AI search and is no longer named as a source there. Since 2023 websites have increasingly blocked AI providers, and in very different ways; researchers often found contradictions between terms of use and robots.txt Longpre et al. 2024. That is why the check shows each crawler separately.
| Type | Entries | What they do |
|---|---|---|
| AI search | 11 | Builds the index from which an AI assistant or a search engine gets answers with sources. If blocked, the page appears there less often as a source, or not at all. |
| Training | 6 | Collects content that language models learn from. A ban affects future training runs, not models that are already trained. |
| Training switch | 2 | Not a crawler of its own. The rule tells the operator whether it may use what it fetches anyway for training. |
| User fetch | 7 | Fetches a single page because a person has just asked about it. Several operators state that robots.txt does not apply to this, or not always. |
Two entries deserve a second look. Common Crawl does not train a model itself, but it builds the largest freely available archive of the web, and that is one of the most important sources for the pre-training of large language models Baack 2024. The well-known training corpus C4, for instance, was built from a single such snapshot Dodge et al. 2021. Anyone who refuses training therefore blocks CCBot too; but no rule takes back what is already in a snapshot. The second is Googlebot: it is on the list because the AI Overviews and AI Mode of Google Search build on its index. Nobody who wants to be found on Google blocks it.
All 27 crawlers covered
| Name in the robots.txt | Operator | Purpose | robots.txt applies | What the operator says |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search | yes, according to the operator | Feeds the search in ChatGPT. If blocked, the site does not appear there as a source in answers. |
| Claude-SearchBot | Anthropic | AI search | yes, according to the operator | Feeds the search in Claude. If blocked, the page appears less often in results there. |
| Googlebot | AI search | yes, according to the operator | The crawler of Google Search. AI Overviews and AI Mode also draw on this index; they are controlled there through nosnippet, max-snippet and noindex. | |
| PerplexityBot | Perplexity | AI search | yes, according to the operator | Builds the index from which Perplexity links sources. According to the operator, it is not used to train models. |
| Applebot | Apple | AI search | yes, according to the operator | Crawler for search in Siri, Spotlight and Safari. According to Apple, the data may also go into the training of Apple’s models; Applebot-Extended exists for that. |
| meta-webindexer | Meta | AI search | yes, according to the operator | Indexes for search in Meta AI so that Meta can cite and link content. |
| Amzn-SearchBot | Amazon | AI search | yes, according to the operator | For search functions in Amazon products such as Alexa; according to Amazon, not for training. |
| bingbot | Microsoft | AI search | yes, according to the operator | The crawler of Bing. Microsoft names no separate AI identifier; from February 2027 the settings for training and AI answers are addressed to Bingbot. |
| MistralAI-Index | Mistral | AI search | yes, according to the operator | Crawls only for the search index; according to Mistral, not for training. |
| DuckAssistBot | DuckDuckGo | AI search | yes, according to the operator | Fetches pages in real time for DuckDuckGo’s AI answers, with source attribution. According to the operator, not for training. |
| YouBot | You.com | AI search | yes, according to the operator | Crawler of the You.com search engine. Follows robots.txt including Crawl-delay. |
| GPTBot | OpenAI | Training | yes, according to the operator | Collects content that may go into the training of OpenAI’s models. According to OpenAI, blocked means: do not use for training. |
| ClaudeBot | Anthropic | Training | yes, according to the operator | Collects content that may go into the training of Anthropic’s models. Anthropic follows robots.txt and Crawl-delay too. |
| meta-externalagent | Meta | Training | yes, according to the operator | Collects for training AI models and for Meta’s products. |
| Amazonbot | Amazon | Training | yes, according to the operator | Improves Amazon’s products, which can include training Amazon’s AI models. It does not honour Crawl-delay. |
| MistralAI-Training | Mistral | Training | yes, according to the operator | Collects content for training data of the Mistral models. |
| CCBot | Common Crawl | Training | yes, according to the operator | Builds a freely accessible archive of the web. Common Crawl trains nothing itself, but the archive is one of the most important sources of training data. |
| Google-Extended | Training switch | yes, according to the operator | Not a crawler of its own but a switch: may Google’s fetches go into the training of future Gemini models and into the grounding of Gemini answers? According to Google, it has no effect on Google Search. | |
| Applebot-Extended | Apple | Training switch | yes, according to the operator | Not a crawler of its own. If blocked, Apple does not use the content for training its models; the page stays in search. |
| ChatGPT-User | OpenAI | User fetch | not always | Fetches a page when someone asks about it in ChatGPT. According to OpenAI, robots.txt rules may not apply to this. |
| Claude-User | Anthropic | User fetch | yes, according to the operator | Retrieves pages when users ask a question in Claude. Anthropic says its bots follow robots.txt. |
| Google-Agent | User fetch | not always | Browses on behalf of a user and carries out actions. According to Google, user fetches usually bypass robots.txt. | |
| Perplexity-User | Perplexity | User fetch | not always | Fetches pages when a user asks. According to Perplexity, it usually ignores robots.txt. |
| meta-externalfetcher | Meta | User fetch | not always | Fetches individual links at the user’s request, also for AI agents. According to Meta, it can bypass robots.txt. |
| Amzn-User | Amazon | User fetch | not always | Fetches current data for a user’s question, for example for Alexa. According to Amazon, it does not follow all robots.txt rules. |
| MistralAI-User | Mistral | User fetch | yes, according to the operator | Fetches a page when a user asks and links it as a source. According to Mistral, the robots.txt group determines which pages qualify. |
| Bytespider | ByteDance | unclear | not documented | No operator documentation found. Purpose and handling of robots.txt are not documented. |
Not included: names from bot directories that the operator itself does not give (such as anthropic-ai or cohere-ai). According to its own documentation, Cohere currently runs no crawler; Microsoft names no AI identifier of its own, its settings for AI are addressed to bingbot. As of 14 November 2026.
03 / GENERATOR
Block AI training, allow AI search: the matching block
Most businesses want the same thing: to appear as a source in the answers of ChatGPT, Claude or Perplexity, but not to supply training material unasked. The generator is preset for that. It writes each group separately and only with Allow and Disallow, without wildcards, because even commercial crawlers repeatedly misread certain rules Giles et al. 2010. You paste the block into your existing robots.txt; the group “User-agent: *” stays as it is.
Generator · robots.txt block
The generator runs in the browser and sends nothing.
Generator · llms.txt outline
Runs in the browser. The file goes into the root directory of the website as llms.txt.
The most common trap: groups inherit nothing
A crawler follows exactly the group with its name. If it exists, the catch-all rule no longer applies to it, not even partly. In the example on the right, “User-agent: *” blocks the area /intern/, but GPTBot gets its own group with “Allow: /”.
The result for /intern/preise: ClaudeBot is blocked (line 2: Disallow: /intern/ (group *)), whereas GPTBot may read (line 5: Allow: / (group GPTBot)). Anything that is to be blocked for everyone therefore belongs in every named group. In the generator you enter such areas once and it repeats them everywhere.
User-agent: * Disallow: /intern/ User-agent: GPTBot Allow: /
The shared training signal
Since 30 October 2026 there has been an IETF draft meant to shorten the blocking of crawlers one by one: a group “User-agent: __usage_ai-training” with “Disallow: /” tells all participating providers at once that you do not want training. Paths work as usual; “Allow: /presse/” exempts the press kit. The group is not a crawling instruction, and the catch-all rule “User-agent: *” expressly does not count for it. Because the draft is young and not every provider reads it yet, the generator writes both: the individual groups and the signal.
| Provider | From when |
|---|---|
| Anthropic | since 30 October 2026; a named ClaudeBot group takes precedence |
| Microsoft (Bing) | from 1 February 2027 |
| planned for 2027; without a group of its own for Google-Extended, Google will then treat a No like a blocked Google-Extended | |
| OpenAI | “in the coming months”; the GPTBot rules remain |
Source: draft-kelley-aipref-training-preference-signal-00, without standard status, expires on 3 May 2027. As of 14 November 2026.
04 / RULES
How a crawler reads your robots.txt
Since September 2022 the procedure has been laid down as RFC 9309, and the check calculates by it too. The file always sits at /robots.txt in the root directory and applies only to exactly that host: if your-company.de redirects to www.your-company.de, the file on www. counts. The crawler looks for the group with its name, upper or lower case does not matter; if there is none, it takes “User-agent: *”, and if there is none of that either, it may read everything. Within the group the longest matching rule wins; if equal, Allow. An asterisk stands for any characters, a dollar sign at the end for the end of the address.
As early as 2007 a survey of 7,593 websites showed that the file is used more and more, and also what it is: a rule that decent crawlers follow, not enforcement Sun et al. 2007. Nothing has changed there.
| Response | RFC 9309 | |
|---|---|---|
| 200, text file | Rules apply, at least 500 KiB are read | likewise, everything after 500 KiB is ignored |
| 200, but a web page | no valid line, so no rule | likewise |
| 404 and other 4xx | no robots.txt, everything allowed | everything allowed, except for 429 |
| 429 too many requests | like 4xx | like a server error |
| 5xx or unreachable | everything blocked | no crawling for 12 hours, then the last good version for up to 30 days |
| Redirect | follow at least five | at least five, then like 404 |
Google usually caches the file for up to 24 hours, Amazon for up to 30 days. Changes therefore do not take effect immediately. Our fetch follows at most three redirects. As of 14 November 2026.
Directives on the pages themselves
Besides the robots.txt there are directives per page, as a meta tag in the head or as an X-Robots-Tag in the server response. The check reads them on the home page. For AI these matter most: noindex takes the page out of the search index and so also out of AI Overviews; nosnippet forbids the text excerpt at Google, including in AI Overviews and AI Mode. Amazon reads noarchive as “not for training”. From 1 February 2027 Microsoft introduces its own directives for training and grounding (ai-training, ai-grounding) and from March 2027 no longer reads noarchive and nocache as an AI directive. Directives such as “noai” are widespread, but not documented by any operator on our list.
05 / LIMITS
What the robots.txt does not do
- It is a request. In trials with 130 named bots over 40 days, bots followed strict rules less often, and AI search crawlers often did not even request the file Kim et al. 2025.
- It only works on names. With specially prepared websites it was possible to show which fetcher feeds which AI system, including several that their operator does not disclose Seiden et al. 2026. Anyone who arrives under a different name is not covered by any rule.
- User fetches partly bypass it. OpenAI, Perplexity, Meta, Amazon and Google themselves state that their fetches at a user’s request do not always follow robots.txt. Anthropic and Mistral name the file as the control for these as well.
- Hard blocks sit on the server. Artists often lacked the knowledge or the access to their own website for robots.txt; blocks at the reverse proxy protected more effectively Liu et al. 2025. Conversely, some hosts and site builders block crawlers without the owner knowing: in one Common Crawl snapshot about 1.68 per cent of websites expressly turned the crawler away Ansar et al. 2025. If your home page answers our fetch with a challenge page, the check reports it.
- Our fetch is not an AI crawler. The server fetches under its own identifier. If a firewall blocks crawlers by name, for instance through a Cloudflare setting, the check does not see it.
And legally?
Under § 44b para. 3 UrhG (German Copyright Act), text and data mining of freely accessible works is permitted unless the rights holder has reserved that use, and for content online only a reservation in machine-readable form counts. The robots.txt is machine-readable, but whether it suffices as a reservation has not been conclusively settled. A legal study argues for common-law countries that ignoring the file can, in some circumstances, give rise to liability Chang and He 2025; for Germany this has to be assessed separately. Anyone who wants to declare a reservation should therefore also put it in the legal notice (Impressum) or in the terms of use. This is not legal advice.
06 / LLMS.TXT
llms.txt: what the file can do and what it cannot
The llms.txt was proposed by Jeremy Howard on 3 September 2024; the description is on llmstxt.org. It is a Markdown file at /llms.txt: a heading with the name, which is the only mandatory part, then a short description as a quotation, optionally a few sentences, and sections with lists of links to the most important pages. A section “Optional” holds what an assistant can leave out.
A measurement of about 4 million domains found the file mainly at tech companies, and some of their links pointed to pages that their own robots.txt blocks Hoffmann et al. 2026. That is exactly what the check tests. A signpost to blocked pages leads nowhere.
To be honest: in its documentation on AI Overviews Google writes that no new machine-readable files are needed for them, and none of the operators covered here says in its crawler documentation that it reads the llms.txt of other websites. So it is not a lever for visibility. It still makes sense as a short, clean self-description that an assistant can read during a user fetch, and because it is done in ten minutes.
Our own is at aureum-tech.com/llms.txt. The generator above builds the outline for yours.
07 / VISIBILITY
Being allowed to read is not yet being named
The check answers a preliminary question. Whether ChatGPT names your company when someone asks for an electrician in Plauen depends on the content of your pages and on what others write about you. In trials, the visibility of sources in AI answers could be raised by up to 40 per cent through how the content was prepared, with different methods depending on the field Aggarwal et al. 2024. At Google, a measurement of 55,393 search queries showed an AI Overview in 13.7 per cent of cases, and for questions in 64.7 per cent, and almost 30 per cent of the domains cited there were not among the hits on the first page Xu et al. 2026.
Named does not mean correctly cited either. In four AI search engines, on average only a little over half of the sentences were backed by the sources given Liu et al. 2023. And anyone who blocks Google’s AI crawler is used less often as a source in AI Overviews, although Google writes that the switch does not affect search Grossman et al. 2026.
The SpotLens quick check shows how visible your company really is in ChatGPT, Perplexity and Google’s AI. How we build content and structured data so that AI searches cite them is covered under AI-SEO. If you would like to look for yourself first, the guide explains how to check your visibility in ChatGPT and what AI visibility for businesses means. You can test individual rules in the robots.txt tester; further tools for being found are under SEO and visibility.
08 / RESEARCH
What the research says
The assessment in the check rests on 15 peer-reviewed papers. The most important findings follow; the full list with review notes is further down.
- Blocks are increasing, and unevenly. Within a year, large parts of the sources behind AI training data were blocked, differing by provider Longpre et al. 2024. That is why the check shows crawlers individually.
- Rules are followed selectively. The stricter the rule, the less often bots followed it Kim et al. 2025. That is why the check measures nothing it cannot measure, and says so.
- Training runs through archives. Common Crawl is one of the most important sources for pre-training Baack 2024, and corpora such as C4 are built from single snapshots Dodge et al. 2021. That is why the check warns if CCBot stays open.
- AI Overviews choose their own sources. Many cited domains are not on page one Xu et al. 2026, and blocked AI crawlers go along with fewer mentions Grossman et al. 2026.
- llms.txt is young and often contradictory. Links in the file sometimes point to blocked pages Hoffmann et al. 2026; the check tests each link in the file.
09 / FOR AI ASSISTANTS
Using this tool from an AI assistant
Assistants such as Claude or ChatGPT that speak the Model Context Protocol (MCP) can call this page’s calculation directly instead of rebuilding it. The server is public, needs no sign-in and stores nothing. The answer always includes the link to this page, the date of the data and the limits of the result.
- Endpoint
https://aureum-tech.com/mcp- Tool
werkzeug_ki_crawler_check- Transport
- Streamable HTTP, read-only
| Name | Required | Meaning |
|---|---|---|
domain | no | Domain of the website, e.g. your-company.de or https://www.your-company.de. A path is ignored, robots.txt always sits in the root directory. Public websites only, port 80 or 443. |
robots_txt | no | Instead of the domain: the text of a robots.txt, which is evaluated without a fetch. Do not give it together with domain. |
pfad | no | Path to check, e.g. /blog/article. Without it, the home page “/”. |
Questions where an assistant should call it
- “May ChatGPT read my website handwerk-muster.de?”
- “Does our robots.txt block the AI search of Perplexity and ChatGPT by mistake?”
- “How do I forbid AI training but stay visible in the search of ChatGPT and Claude?”
- “Does www.beispiel-praxis.de have an llms.txt, and do its links point to blocked pages?”
How the connection works and what MCP can do in your own business is explained on the page about MCP integrations.
10 / COMMON QUESTIONS
Common questions about AI crawlers, robots.txt and llms.txt
How do I block AI training without dropping out of ChatGPT search?
OpenAI keeps the two apart: GPTBot collects for training, OAI-SearchBot for search in ChatGPT, and each setting works on its own. In the robots.txt, GPTBot therefore gets “Disallow: /” and OAI-SearchBot gets “Allow: /”. The same applies to ClaudeBot and Claude-SearchBot at Anthropic; at Google you block Google-Extended and leave Googlebot alone. The generator on this page writes the whole block for all providers covered.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects content that may go into the training of future models. OAI-SearchBot builds the index from which ChatGPT names sources in a search; according to OpenAI, a site that blocks it does not appear in answers there. ChatGPT-User fetches a single page because someone in the chat has just asked about it. According to OpenAI, robots.txt rules may not apply to this kind of fetch.
Will my website drop out of Google if I block Google-Extended?
No. Google-Extended is not a crawler of its own but a switch for whether Google may use what it has fetched for Gemini training and for grounding Gemini answers. According to Google it affects neither inclusion in search nor ranking. The AI Overviews in Google Search depend on Googlebot and on snippet rules such as nosnippet. A 2026 measurement did find, however, that pages blocking Google’s AI crawler were cited less often as a source in AI Overviews.
Do I need an llms.txt?
It is not mandatory. The llms.txt is a proposal by Jeremy Howard from September 2024: a Markdown file with a heading, a short description and links to the most important pages. Google states explicitly that AI Overviews need no new machine-readable files, and none of the operators covered here says in its crawler documentation that it reads it. It does no harm and is quickly done; the generator provides the outline. More important is that its links do not point to blocked pages.
Does every crawler follow the robots.txt?
No. The robots.txt is a request, not a lock. Measurements show that bots follow stricter rules less often and that AI search crawlers often do not request the file at all. According to their operators, user fetches such as ChatGPT-User, Perplexity-User or meta-externalfetcher partly bypass the robots.txt, and anyone who collects without giving a name appears in no rule. If you really want to block, you need a block on the server or at your host.
Does a block also apply to models that are already trained?
No. OpenAI and Anthropic describe the block as a signal for future training. Training data comes from fixed snapshots of the web, such as the Common Crawl archive; what is already there stays there. That is why it is worth including CCBot if you want to refuse training.
What does “User-agent: __usage_ai-training” mean?
It is the shared training signal from an IETF draft of 30 October 2026. A group with this name and “Disallow: /” tells all participating providers at once that you do not want training. Anthropic says it already honours it; Microsoft will do so from 1 February 2027, Google plans to honour it in 2027, and OpenAI in the coming months. It is not a crawling instruction, and the group “User-agent: *” expressly does not count for it.
Where does the AI Crawler Check stop?
It reads the rules of your robots.txt, the llms.txt and the robots directives on the home page, and tells you per crawler what is allowed. It does not measure whether a crawler complies, nor whether an AI assistant knows and recommends your company. The SpotLens quick check shows that; how content has to be built so that AI searches cite it is covered under AI-SEO. We can set the files up for you on request.
11 / STUDIES AND SOURCES
What the tool is based on
The guide values and notes above are based on these 15 scientific papers. Each one has been checked via its DOI or the publisher’s page (most recently on 14 November 2026): it exists, it has not been retracted, and its findings still hold for this question today.
Sun, Y.; Zhuang, Z.; Giles, C. L. (2007): A large-scale study of robots.txt. Proceedings of the 16th International Conference on World Wide Web (WWW 2007), pp. 1123–1124
Finding Five surveys of 7,593 websites show that robots.txt is used more and more. The file is not enforcement but a rule that decent crawlers follow.
In the tool Sets out the page’s principle: robots.txt is a request to crawlers, not a lock.
doi.org/10.1145/1242572.1242726 · Conference paper
Giles, C. L.; Sun, Y.; Councill, I. G. (2010): Measuring the web crawler ethics. Proceedings of the 19th International Conference on World Wide Web (WWW 2010), pp. 1101–1102
Finding Most commercial crawlers behaved in line with the rules, but many repeatedly violated or misunderstood certain robots.txt rules.
In the tool That is why the generator writes each group separately and only with Allow and Disallow, without wildcards.
doi.org/10.1145/1772690.1772824 · Conference paper
Kim, T.; Bock, K.; Luo, C. et al. (2025): Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. Proceedings of the 2025 ACM Internet Measurement Conference (IMC 2025), pp. 541–557
Finding In controlled trials with 130 named bots over 40 days, bots followed stricter rules less often; AI search crawlers often did not request the robots.txt at all.
In the tool Underpins the limit “read rules, do not measure behaviour” and the note on blocked user fetches.
doi.org/10.1145/3730567.3764471 · Conference paper
Liu, E.; Luo, E.; Shan, S. et al. (2025): Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. Proceedings of the 2025 ACM Internet Measurement Conference (IMC 2025), pp. 78–99
Finding Measurements and a survey of 203 artists show great demand for robots.txt, but hurdles in knowledge and in access to their own website. Blocks at the reverse proxy protected more effectively but are not widespread.
In the tool Hence the recommendation to involve the host or proxy for a hard block, and the generator, which presumes no technical vocabulary.
doi.org/10.1145/3730567.3732913 · Conference paper
Seiden, S.; Ren, T.; Zhang, C. et al. (2026): Identifying AI Web Scrapers Using Canary Tokens. Proceedings of the 2026 ACM SIGSAC Conference on Computer and Communications Security (CCS 2026), pp. 6998–7012
Finding With specially prepared websites and 22 AI systems it was possible to determine which fetcher feeds which model, including several that their operator does not disclose.
In the tool Supports the limitation that the table only knows crawlers that their operator documents under its own name.
doi.org/10.1145/3830454.3846799 · Conference paper
Longpre, S.; Mahari, R.; Lee, A. et al. (2024): Consent in Crisis: The Rapid Decline of the AI Data Commons. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pp. 108042–108087
Finding An audit of 14,000 domains behind AI training data found, within a year, a sharp rise in blocks, large differences between AI providers and frequent contradictions between terms of use and robots.txt.
In the tool That is why the check shows each crawler individually instead of merely reporting “AI allowed” or “AI blocked”.
doi.org/10.52202/079017-3431 · Conference paper
Baack, S. (2024): A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024), pp. 2199–2208
Finding Common Crawl is the largest freely available collection of web data and one of the most important sources for the pre-training of large language models, often with a large share of the training material.
In the tool That is why CCBot is listed under training, and the check warns if individual training crawlers are blocked but CCBot is not.
doi.org/10.1145/3630106.3659033 · Conference paper
Dodge, J.; Sap, M.; Marasović, A. et al. (2021): Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pp. 1286–1305
Finding The training corpus C4 was built from a single Common Crawl snapshot and contains text from unexpected sources such as patent documents and military sites.
In the tool Shows that training corpora are built from fixed snapshots: what is once in there is not reached by a later ban.
doi.org/10.18653/v1/2021.emnlp-main.98 · Conference paper
Aggarwal, P.; Murahari, V.; Rajpurohit, T. et al. (2024): GEO: Generative Engine Optimization. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2024), pp. 5–16
Finding Content can be prepared so that AI search engines cite it up to 40 per cent more visibly; what works depends on the field.
In the tool Draws the distinction: access is the precondition, the content decides whether it is cited. For that the page points to AI-SEO.
doi.org/10.1145/3637528.3671900 · Conference paper
Liu, N.; Zhang, T.; Liang, P. (2023): Evaluating Verifiability in Generative Search Engines. Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001–7025
Finding In four AI search engines, on average only a little over half of the sentences were fully backed by sources, and about a quarter of the citations did not support the sentence.
In the tool States the limitation that free access does not promise a correct citation: the check tests reading, not the answer.
doi.org/10.18653/v1/2023.findings-emnlp.467 · Conference paper
Xu, H.; Iqbal, U.; Montgomery, J. M. (2026): Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact. Proceedings of the 2026 ACM Internet Measurement Conference (IMC 2026), pp. 529–546
Finding Across 55,393 search queries, an AI Overview appeared in 13.7 per cent of cases, and for questions in 64.7 per cent. Almost 30 per cent of the cited domains were not among the hits on the first page.
In the tool Explains why Googlebot is in the check: AI Overviews choose sources from the index, but not only from the top hits.
doi.org/10.1145/3777912.3839818 · Conference paper
Grossman, R.; Liu, S.; Chen, M. K. et al. (2026): How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2026), pp. 448–459
Finding Across 11,500 search queries, the sources of Google Search, AI Overviews and Gemini differed greatly. Websites that block Google’s AI crawler were used less often as a source in AI Overviews.
In the tool The note on a blocked Google-Extended sets this finding beside Google’s statement that the switch does not affect search.
doi.org/10.1145/3805712.3809667 · Conference paper
Hoffmann, F.; Goergens, T.; Khosla, S.; Bajpai, V. (2026): From robots.txt to ai.txt: Mapping the Evolution of Web Permissions in the Age of AI. ACM SIGCOMM Computer Communication Review 56, pp. 29–41
Finding A measurement of about 4 million domains shows: robots.txt remains the most important tool; AI-specific files such as ai.txt and llms.txt appear mainly at tech companies. Some links in llms.txt files point to pages that the robots.txt blocks.
In the tool Hence the test of whether links in the llms.txt point to blocked pages.
doi.org/10.1145/3831956.3831960 · Journal article
Chang, C.-Y.; He, X. (2025): The liabilities of robots.txt. Computer Law & Security Review 58, article 106176
Finding With generative AI, ignoring robots.txt weighs more heavily. The paper argues for common-law countries that the file can, in some circumstances, work as a unilateral contract or as a warning with consequences for liability.
In the tool Supports the note that robots.txt is more than technology, with the proviso that the legal position in Germany has to be assessed separately.
doi.org/10.1016/j.clsr.2025.106176 · Journal article
Ansar, M.; Sperotto, A.; Holz, R. (2025): Web Crawl Refusals: Insights From Common Crawl. Passive and Active Measurement (PAM 2025), Lecture Notes in Computer Science, pp. 197–214
Finding About 1.68 per cent of websites in a Common Crawl snapshot expressly turned the crawler away, often through large hosts and site-builder providers, ranging from a block to a challenge page.
In the tool Hence the note when the home page answers with a challenge page: bot protection blocks independently of the robots.txt.
doi.org/10.1007/978-3-031-85960-1_9 · Conference paper
Standards, authorities and manufacturer data
- RFC 9309: Robots Exclusion Protocol · IETF, September 2022
- How Google interprets the robots.txt specification (status codes, 500 KiB, caching) · Google Search Central
- Overview of OpenAI Crawlers · OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic
- Google common crawlers (Google-Extended) · Google Search Central
- Google user-triggered fetchers (Google-Agent) · Google Search Central
- AI features and your website · Google Search Central
- Perplexity Crawlers · Perplexity
- About Applebot · Apple, as of 4 September 2026
- Meta Web Crawlers · Meta
- Amazonbot, Amzn-SearchBot, Amzn-User · Amazon, as of 26 October 2026
- Your content, your choice: clearer preferences for training and grounding · Microsoft Bing, 30 October 2026
- Mistral AI crawlers · Mistral AI
- DuckAssistBot · DuckDuckGo
- YouBot · You.com
- CCBot · Common Crawl
- Cohere web crawlers (currently no crawler) · Cohere
- draft-kelley-aipref-training-preference-signal-00 · IETF draft, 30 October 2026
- A new way to control AI training (training signal) · Google Search Central Blog, 30 October 2026
- Making it easier for websites to express AI training preferences · OpenAI, 30 October 2026
- The /llms.txt file · llmstxt.org, proposal of 3 September 2024
- § 44b UrhG: text and data mining · gesetze-im-internet.de
12 / MORE TOOLS
Still have a question?
- AT-W2 · Home & business
Windows 10 check
Does this PC still get security updates, and what needs doing by October 2027? - AT-W3 · Home & business
Wi-Fi planner
How many access points or mesh units does your home, office or warehouse need? - AT-W4 · Businesses
IT emergency check
How quickly could your business get back to work after an outage? - AT-W5 · Businesses
E-mail check
Are your e-mails landing in spam? Your domain’s SPF, DKIM and DMARC in plain language. - AT-W6 · Businesses
UPS calculator
Which UPS keeps your server, router and till running long enough during a power cut? - AT-W8 · Businesses
XRechnung viewer
Received an XRechnung as an XML file? This makes it readable without the file leaving your computer.
Send us the copied result. We set up robots.txt, llms.txt and structured data so that AI searches read your pages and training only takes place where you want it, in Rodewisch and across the whole Vogtland.
+49 3744 4399760[email protected]Mon–Fri 8:00–17:00 · Straße der Jugend 5, 08228 Rodewisch