Robots.txt: What It Controls, What It Doesn't

On September 14, 2026, we typed “robots.txt” into Google in Brazil, where we are based. The first result was Google’s own page on the file. ¹ In its very first paragraph, that page warns that robots.txt is no way to keep a page out of Google. ²

Scroll down the same results page. Among Brazilian texts from e-commerce platforms and web hosts, the sentence flips. One says the file controls what search engines will index. Another teaches that it is how you stop a page from appearing on Google. ³ I read five of these texts. Four say the same thing in different words. ³

None of them was written in bad faith. The name misleads: blocking the crawler sounds like blocking Google itself. This piece separates the two, with the specification and the documentation open on September 14, 2026.

What robots.txt is

Think of the lobby of an office building. On the wall, a sign says delivery drivers may not go up to the third floor. A polite driver reads the sign and complies. The sign locks no door. Whoever decides to ignore it goes up anyway.

A site has that sign at its entrance. It is a plain text file, always in the same place: the root of the site, followed by /robots.txt. ⁴ Before visiting the pages, Google’s crawler downloads this file and reads what it may visit. ⁵ This is called the Robots Exclusion Protocol, and robots.txt is the sign where it gets written.

The rule was born in 1994, proposed by the engineer Martijn Koster. It got a formal specification at the IETF, the body that publishes the internet’s technical standards, in September 2022. It is RFC 9309, written by Koster and three Google engineers. ⁶ The RFC says these rules are not a form of access authorization. ⁶ A sign in the lobby, with no lock.

One address detail catches a lot of people. The file applies only to the host, protocol, and port where it is published. ⁵ The robots.txt on example.com does not apply to shop.example.com, or to the http version of the same site. ⁵ Each subdomain needs its own. And a file inside a folder does not count, because crawlers do not look for robots.txt in subdirectories. ⁵

The four lines Google reads

The file is made of blocks. Each block starts by naming the crawler it addresses, followed by the rules for that crawler. This is called a group. ⁴

Google supports four fields. ⁵

User-agent: the name of the crawler the group addresses.
Disallow: a path it should not visit.
Allow: a path it may visit, even inside a blocked area.
Sitemap: the full address of the sitemap.

Google ignores the rest. Crawl-delay, which many people copy from ready-made templates, is not on its list. ⁵ Field names can be uppercase or lowercase. Paths cannot. To a crawler, /Cart/ and /cart/ are different addresses. ⁵

Two wildcards round out the syntax. The asterisk stands for any sequence of characters. The dollar sign marks the end of the address. ⁵ Following the documentation’s logic, Disallow: /*.pdf$ blocks addresses that end in .pdf, and only those. ⁵

When two rules match the same address, the more specific one wins, the one with the longer path. ⁵ If they tie, Google goes with the least restrictive one, the one that allows. ⁵ The RFC calls for the same thing. ⁶

The most common trap lives in the groups. Each crawler follows one group only, the one with its name. ⁵ If there is a group for Googlebot, it sets aside the rules in the asterisk group. ⁵ Whoever writes a group for a crawler has to repeat there everything they want it to respect.

The asterisk also has an exception at Google. It does not reach the AdsBot crawlers, which must be named explicitly. ⁴ That crawler showed up in the Search Console crawl data of a site that runs no ads, in the piece on crawling and crawl budget. That piece also covers Google’s crawler categories and which of them obey the file.

An example, line by line

The file below is made up for this piece. The domain example.com is reserved for documentation and belongs to no one. ⁷ The paths are in Portuguese because this is the exact file we checked. Carrinho means cart, ordem means sort order, and the first line is a comment saying the example is made up.

bloco para copiar

# Exemplo inventado para este texto
User-agent: *
Disallow: /carrinho/
Disallow: /*?ordem=
Allow: /carrinho/como-comprar.html

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

The first group applies to every crawler without a group of its own. It asks them not to visit the cart folder. It also asks them to skip every address with ?ordem=, the product list re-sorted by price or name. Inside the cart, it allows a single page, the help page on how to buy. Its path is longer, so it wins. ⁵

The second group applies to GPTBot, the crawler OpenAI uses to collect content that may go into training. It stays out of the entire site. ⁸ Since it has a group of its own, GPTBot does not read the first one. ⁵ OAI-SearchBot, the same company’s search crawler, has no group here. It follows the asterisk group and may visit the products. ⁵ ⁸

That leaves the sitemap line, with the full address. It belongs to no group and applies to every crawler. ⁵ I ran the example through a script that applies the documentation’s precedence rules, and each address landed where this paragraph says. ⁹

What robots.txt does not do

Go back to the lobby. The sign tells the driver where not to go. It does not take the store’s address out of the city guide.

A web page works the same way. A page blocked in robots.txt can still get into Google if other sites link to it. ² Google does not read the content. It may still keep the address and the text of the links pointing to it. ² In that case, the result shows up with no description. ² Search Console’s page indexing report has a status just for this, called “Indexed, though blocked by robots.txt.” ¹⁰

To take a page out of Google, the documentation points to three routes: the noindex rule, a password, or deleting the page. ² And this is where a trap catches a lot of people.

In the piece on a site that does not show up on Google, noindex got the image of a sign in a storefront window: no photos. That sign hangs in the shop’s own window, upstairs. If the lobby sign says not to go up, the photographer never gets to read it.

The documentation spells this out. For noindex to work, the page must not be blocked in robots.txt. ¹¹ If it is blocked, the crawler never sees the rule, and the page can still appear in results. ¹¹ Google also does not support noindex written inside robots.txt. ¹¹ The WordPress checkbox that hides a site while it is being built is explained in the same piece.

The file protects nothing, either. The RFC says it is no substitute for security measures. ⁶ And it warns of the opposite effect: listing a path in the file makes that path public and easy to find. ⁶ A restricted area calls for a password on the server. ²

Not every crawler obeys. Google writes that the file forces no one, and that it is up to the crawler to follow it. ² Googlebot and other respectable crawlers follow it. Others might not. ²

When the file fails

The file can vanish, break, or grow too large. Google documents what it does in each case. ⁵

No file, or a 4xx error other than 429. Google acts as if there were no restrictions at all. ⁵
Server errors, the 5xx. For the first 12 hours, Google stops crawling the site and keeps trying to fetch the file. After that, for up to 30 days, it uses the last good version it has. ⁵
An error that lasts more than 30 days. If the site is up, Google acts as if there were no file. If the site has a general availability problem, Google stops crawling. ⁵
Redirects. Google follows at least five hops. After that, it treats the file as missing. ⁵
A web page instead of the file. If the address returns HTML, Google tries to pull valid rules from it and ignores the rest. ⁵
Size. The limit is 500 KiB, a little over half a megabyte. Anything beyond that is ignored. ⁵ The RFC requires crawlers to read at least that much. ⁶

The first item surprises people who expect the opposite. A file that vanishes opens everything. A server that fails to deliver the file freezes crawling of the whole site for half a day.

There is also memory. Google generally caches the file for up to 24 hours. ⁵ A fix made now can take a day to take effect.

AI crawlers, by what each company wrote

Here the file turned into a business decision. Each AI company runs different crawlers for different jobs, with published names. In my book SEO de Entidade, published in Portuguese, I call this three robots with three badges. ¹² The table shows only what each company’s official documentation says, checked on September 14, 2026.

Company	Name in robots.txt	What the documentation says
Google	`Googlebot`	crawls for Search, including its AI features ¹³ ¹⁴
Google	`Google-Extended`	a control token with no crawler of its own; decides whether content trains Gemini models and grounds answers in the Gemini apps and Vertex AI; does not affect inclusion in Search ¹³
OpenAI	`OAI-SearchBot`	surfaces sites in ChatGPT search; a blocked site drops out of search answers ⁸
OpenAI	`GPTBot`	collects content that may train the models; blocking it signals that the content stays out of training ⁸
OpenAI	`ChatGPT-User`	visits the page at a user’s request; robots.txt may not apply ⁸
Perplexity	`PerplexityBot`	surfaces and links sites in Perplexity search; does not collect for training ¹⁵
Perplexity	`Perplexity-User`	visits the page at a user’s request; generally ignores robots.txt ¹⁵
Anthropic	`ClaudeBot`	collects content that may go into training; blocking it removes future material from training ¹⁶
Anthropic	`Claude-SearchBot`	indexes for search; blocking it may reduce visibility in answers ¹⁶
Anthropic	`Claude-User`	visits the page at a user’s request; blocking it may reduce visibility ¹⁶

Google asks for a careful reading. AI features in Search, such as AI Overviews and AI Mode, are part of Search. That is why the access control for them is the robots.txt rule for Googlebot. ¹⁴ Google-Extended does not take a site out of Search, and Google writes that it is no ranking signal. ¹³ To limit what appears in those features, Google points to snippet rules, such as nosnippet and max-snippet, and to noindex. ¹⁴

There is also Search Console’s generative AI control, with a cost written into its own help page. ¹⁷ That control, and the rest of the path to a citation, are in the piece on showing up in AI answers.

At the three AI companies in the table, training and search have separate crawlers. OpenAI writes that a site can allow the search crawler and block the training one. ⁸ OpenAI and Anthropic say what happens to a site that blocks the search crawler: it drops out of answers, or loses visibility. ⁸ ¹⁶ Perplexity recommends allowing PerplexityBot to appear. ¹⁵

Crawlers triggered by a user’s question follow a different rule. OpenAI and Perplexity warn that these may not obey the file. ⁸ ¹⁵ Anthropic says its bots respect robots.txt and support Crawl-delay, the field Google ignores. ¹⁶ ⁵

The block nobody saw in WordPress

A case from our own work, without the client’s name. The team edited robots.txt through WordPress, with the site’s rules and a note for AI crawlers. ¹⁸ Anyone who opened /robots.txt in a browser saw something else. Before the team’s rules, there was a block nobody there had written. ¹⁸ It closed the entire site to names tied to AI training, among them GPTBot, ClaudeBot, and Google-Extended. ¹⁸

The block came from the CDN, the network that delivers the site to visitors. In this case, Cloudflare. Its documentation describes the mechanism. With the setting turned on in the dashboard, it places a block of its own before the site’s robots.txt. And it serves both as a single response. ¹⁹ The WordPress screen showed only the half the team wrote. ¹⁸

On August 2, 2026, the block was still active. It was on for four of the same owner’s six domains. On the other two, there was no block at all. ¹⁸

Reading it takes the table above. OAI-SearchBot, PerplexityBot, and Claude-SearchBot were not on the list. ¹⁸ By each company’s documentation, their search stayed open. ⁸ ¹⁵ ¹⁶ Google-Extended was on it. By Google’s documentation, that took the content out of Gemini training and out of grounding in the Gemini apps. ¹³

It may have been a deliberate choice by whoever had access to the dashboard. The problem was visibility. The decision lived in a place the marketing team never opened, and the file Google read was a different one.

The practical rule follows from this. Check the file the site actually serves, in the browser, and distrust the system’s screen. Google says to test it this way, by opening the file’s address in a private browsing window. ⁴ What WordPress itself puts in the file is in the piece on SEO for WordPress.

The robots.txt report in Search Console

Search Console shows the file the way Google downloaded it. The robots.txt report lists the files found for the site’s top 20 hosts. ²⁰ For each one, it shows the address, the status of the last fetch, the date, the size, and any parsing issues. ²⁰

It has three practical uses. The first is viewing the last version fetched, with errors and warnings flagged. ²⁰ The second is opening the versions from the last 30 days, useful for dating a change nobody announced. ²⁰ The third is requesting a recrawl of the file after you fix an error. ²⁰ That request does not guarantee that the unblocked pages get crawled right away. ²⁰

Two conditions limit the report. It exists only for domain-level properties, or URL-prefix properties with no folder. ²⁰ And it does not test page by page. To find out whether a specific address is blocked, the tool is URL Inspection. ²⁰

A heads-up for Cloudflare users. The company itself writes that Search Console may flag the content signals line it adds as “Syntax not understood.” It says it has seen no effect on crawling. ¹⁹

The ten-minute check

Have your site’s address and your Search Console access at hand.

Open the file the site serves. Type the domain followed by /robots.txt in a private browsing window. ⁴
Look for blocks nobody on the team wrote. Cloudflare, for example, marks the start and end of its block with comments. ¹⁹
Read group by group. Each crawler with a group of its own sets aside the asterisk group. ⁵
Look for a Disallow: / under the asterisk. It closes the whole site to every crawler without its own group. ⁵
Check the sitemap line. The address must be complete, with the protocol. ⁴
Open the robots.txt report. Look at the status, the warnings, and the versions from the last 30 days. ²⁰
Inspect the page that matters most. URL Inspection tells you whether it is blocked. ²⁰
Decide AI crawler by AI crawler. Use the table in this piece, and write down who decided, and on what date.

The rest of technical SEO, from indexing to speed, is in the technical SEO section.

Two questions before you touch the file

From now on, when someone says “just block it in robots.txt,” you will ask two questions. Block what: the crawler’s visit, or the page’s appearance on Google? And which crawler, from which company, for which job?

The first separates robots.txt from noindex. The second separates training from search.

Open your site’s address followed by /robots.txt now, and read the whole file, line by line.

Frequently asked questions

Does robots.txt take a page out of Google?

Not reliably. A blocked page can still appear, with no description, if other sites link to it. ² To take it out of Google, the documentation points to noindex, a password, or deleting the page. ² And noindex only works if the crawler can read the page. ¹¹

Does every site need a robots.txt file?

No. The Search Console help says not having the file is fine, and that Google then understands it can crawl the whole site. ²⁰ The file is useful when there is something to organize, such as areas with no value for search. ²

Does blocking GPTBot take my site out of ChatGPT?

Per OpenAI’s documentation, GPTBot is the training crawler. ChatGPT search depends on OAI-SearchBot, and a site can allow one and block the other. ⁸

Does blocking Google-Extended take my site out of AI Overviews?

Per Google’s documentation, Google-Extended does not affect inclusion in Search. ¹³ For AI features in Search, the access control is the robots.txt rule for Googlebot. ¹⁴

What is the maximum size of a robots.txt file?

Google reads up to 500 KiB. Anything beyond that is ignored. ⁵

How long does a change take to apply?

Google generally caches the file for up to 24 hours. ⁵ OpenAI mentions about 24 hours for its search to reflect the change. ⁸ Perplexity says up to 24 hours. ¹⁵

Notes and sources

Live reading of the Google Brazil results page for “robots.txt,” September 14, 2026, at https://www.google.com.br/search?q=robots.txt&hl=pt-BR&gl=br&pws=0, in the author’s own browser tab. The search ran on Google in Brazil, where we are based, with the interface in Portuguese. First organic result: “Introdução e guia do robots.txt” (Introduction to robots.txt), from Google for Developers, marked as translated by Google. Organic results on the first page, in order: Google for Developers, Cloudflare, English-language Wikipedia, robotstxt.org, the Portuguese version of Google’s page, MDN, Tray, King Host, and HostMídia, with an RD Station ad at the end. The AI Overview (labeled “Visão geral criada por IA”) said, under “O que ele não faz” (What it doesn’t do): “Não bloqueia a indexação: Um arquivo robots.txt não impede que o Google mostre uma página nos resultados de pesquisa.” (It does not block indexing: a robots.txt file does not stop Google from showing a page in search results.) Accessed September 14, 2026.
Google Search Central, “Introduction to robots.txt,” https://developers.google.com/search/docs/crawling-indexing/robots/intro, page last updated December 10, 2025; Portuguese version, “Introdução ao robots.txt,” last updated December 18, 2025. Accessed September 14, 2026. Passages: “This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page”; “If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won’t have a description”; “we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results”; “password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely”; “it’s up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not”; and the use to “avoid crawling unimportant or similar pages on your site”.
SEO Genome survey, September 14, 2026, of the five Brazilian texts from e-commerce platforms and web hosts that appear on the first page in both readings of the search “robots.txt” (the one in note 1 and Semrush’s, Brazil database, the same day), downloaded and read the same day; all are in Portuguese. King Host, “Entendendo o arquivo robots.txt” (Understanding the robots.txt file), published April 20, 2015 and updated June 17, 2021, https://king.host/wiki/artigo/entendendo-o-arquivo-robots-txt/: “ele controla qual informação do site será ou não indexada (encontrada) nos mecanismos de busca” (it controls which information on the site will or will not be indexed, that is, found, by search engines). HostGator, “Como usar o arquivo robots.txt” (How to use the robots.txt file), undated, https://suporte.hostgator.com.br/hc/pt-br/articles/30820167169811-Como-usar-o-arquivo-robots-txt: “O arquivo robots.txt diz o que deve ou não ser indexado nos motores de busca” (The robots.txt file says what should or should not be indexed by search engines). Tray, “O que é o robots.txt?” (What is robots.txt?), updated August 31, 2026, https://basedeconhecimento.tray.com.br/hc/pt-br/articles/7114295899547-O-que-%C3%A9-o-robots-txt: “se deseja que a página X e Y não sejam indexadas e apresentadas nos sites buscadores como o Google, será através do arquivo robots.txt” (if you want pages X and Y not to be indexed and shown on search engines such as Google, it will be through the robots.txt file). Nuvemshop, March 26, 2026, https://atendimento.nuvemshop.com.br/pt_BR/ferramentas-do-google/como-funciona-o-arquivo-robotstxt-na-nuvemshop: the file indicates “quais páginas devem ou não ser indexadas e exibidas nos resultados de pesquisa” (which pages should or should not be indexed and shown in search results). The exception is HostMídia, undated, https://www.hostmidia.com.br/blog/importancia-robots-txt/, which notes that the file is not a security mechanism and that an external link can lead to a blocked page being indexed. None of the five cites RFC 9309, AI crawlers, the size limit, or the behavior on server errors. Accessed September 14, 2026.
Google Crawling Infrastructure, “How to write and submit a robots.txt file,” https://developers.google.com/crawling/docs/robots-txt/create-robots-txt, page last updated November 21, 2025. The old address, in the Search Central documentation, redirects here. Accessed September 14, 2026. Passages: “A robots.txt file lives at the root of your site. So, for site www.example.com, the robots.txt file lives at www.example.com/robots.txt”; “Your site can have only one robots.txt file”; “Each group begins with a User-agent line that specifies the target of the groups”; “Using an asterisk (*) matches all crawlers except the various AdsBot crawlers, which must be named explicitly”; “The sitemap URL must be a fully-qualified URL”; and the publishing test: “open a private browsing window (or equivalent) in your browser and navigate to the location of the robots.txt file”.
Google Crawling Infrastructure, “How Google interprets the robots.txt specification,” https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec, page last updated August 31, 2026; Portuguese version, “Como o Google interpreta a especificação de robots.txt,” last updated September 12, 2026. The old address, https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt, redirects here. Accessed September 14, 2026. Passages: “before crawling a site, Google’s crawlers download and parse the site’s robots.txt file”; “The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted”; “Crawlers don’t check for robots.txt files in subdirectories”; the fields “user-agent”, “allow”, “disallow”, and “sitemap”, with “other fields such as crawl-delay aren’t supported”; field names not case-sensitive and paths case-sensitive; ”* designates 0 or more instances of any valid character” and ”$ designates the end of the URL”; “crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule”; “Only one group is valid for a particular crawler” and “User agent specific groups and global groups (*) are not combined”; the sitemap “isn’t tied to any specific user agent”; 4xx errors, except 429, treated “as if a valid robots.txt file didn’t exist”; for 5xx errors, “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file”, then “for the next 30 days Google will use the last good version”, and the behavior after 30 days; “Google follows at least five redirect hops”; “if the content downloaded is HTML instead of robots.txt rules, Google will try to parse the content and extract rules, and ignore everything else”; “Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored”; and “Google generally caches the contents of robots.txt file for up to 24 hours”.
M. Koster, G. Illyes, H. Zeller, and L. Sassman, “Robots Exclusion Protocol,” RFC 9309, IETF, September 2022, Standards Track, https://www.rfc-editor.org/rfc/rfc9309.html. Illyes, Zeller, and Sassman sign for Google. Accessed September 14, 2026. Passages: the protocol “originally defined by Martijn Koster in 1994”; “These rules are not a form of access authorization.”; in section 2.2.2, “The most specific match found MUST be used” and, with equivalent rules, the allow rule “SHOULD be used”; in section 2.5, “The parsing limit MUST be at least 500 kibibytes”; and in section 3, “The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable.”
IANA, “Example Domains,” https://www.iana.org/help/example-domains. Accessed September 14, 2026. Domains such as example.com are maintained for use in documentation and are not available for registration or transfer, based on RFC 2606 and RFC 6761.
OpenAI, “Overview of OpenAI Crawlers,” https://developers.openai.com/api/docs/bots. Accessed September 14, 2026. Passages: “OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links”; on GPTBot, “It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”; the example of a site owner who allows OAI-SearchBot and blocks GPTBot; on ChatGPT-User, “Because these actions are initiated by a user, robots.txt rules may not apply”; and “it can take ~24 hours from a site’s robots.txt update for our systems to adjust”.
SEO Genome record, September 14, 2026. The example was checked by our own script that applies the group and precedence rules in note 5 (the group with the crawler’s name or, failing that, the asterisk group; the longest path rule; ties go to the allow rule). Result: Googlebot allowed on /produtos/camisa-azul (products, blue shirt) and on /carrinho/como-comprar.html (cart, how to buy), blocked on /carrinho/ and on /produtos?ordem=preco (products sorted by price); GPTBot blocked across the site; OAI-SearchBot, with no group of its own, allowed on /produtos/camisa-azul and blocked on /carrinho/. The script is not Google’s open-source library.
Search Console Help, “Page indexing report,” https://support.google.com/webmasters/answer/7440203. Accessed September 14, 2026. Under the status “Indexed, though blocked by robots.txt”: “The page was indexed despite being blocked by your website’s robots.txt file. Google always respects robots.txt, but this doesn’t necessarily prevent indexing if someone else links to your page.” The Portuguese version carries an AI translation notice, which is why the Portuguese edition of this piece keeps the status name in English.
Google Search Central, “Block Search indexing with noindex,” https://developers.google.com/search/docs/crawling-indexing/block-indexing, page last updated December 10, 2025. Accessed September 14, 2026. Passages: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file”; “If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results”; and “Specifying the noindex rule in the robots.txt file is not supported by Google.”
Daniel Sócrates, SEO de Entidade (published in Portuguese), 2026, chapter 9, “Para uma IA te citar, você atravessa quatro portões” (For an AI to cite you, you pass through four gates), section “Três robôs com três crachás” (Three robots with three badges). The image of the badges, and the reading that allowing or blocking each crawler has become a distribution decision that should not sit only with whoever runs the server.
Google Crawling Infrastructure, “List of Google’s common crawlers,” https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers, page last updated July 14, 2026. The old address, in the Search Central documentation, redirects here. Accessed September 14, 2026. Passages: common crawlers “always obey robots.txt rules when crawling automatically”; Googlebot affects “Google Search (including Discover and all Google Search features)”; Google-Extended “doesn’t have a separate HTTP request user agent string” and “the robots.txt user-agent token is used in a control capacity”; it decides whether content “may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding” in the Gemini apps and in Vertex AI’s Grounding with Google Search; and “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”
Google Search Central, “AI features and your website,” https://developers.google.com/search/docs/appearance/ai-features, page last updated December 10, 2025. Accessed September 14, 2026. Passages: “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search”; and “To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls. To limit AI training and grounding in some of Google’s other systems, read more about Google-Extended.”
Perplexity, “Perplexity Crawlers,” https://docs.perplexity.ai/docs/resources/perplexity-crawlers. The old address, https://docs.perplexity.ai/guides/bots, redirects here. Accessed September 14, 2026. Passages: “PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models”; “To ensure your site appears in search results, we recommend allowing PerplexityBot”; on Perplexity-User, “Since a user requested the fetch, this fetcher generally ignores robots.txt rules”; and “it may take up to 24 hours for our systems to reflect changes”.
Anthropic, help center, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, April 7, 2026, https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. Accessed September 14, 2026. Passages: ClaudeBot collects content “that could potentially contribute to their training”; “When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets”; “Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results”; restricting Claude-User “may reduce your site’s visibility for user-directed web search”; “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt”; and “we support the non-standard Crawl-delay extension to robots.txt”.
Search Console Help, “Search generative AI control,” https://support.google.com/webmasters/answer/16908024. Accessed September 14, 2026. The control covers AI Overviews, AI Mode, and generative AI features in Discover, and sits in Search Console Settings. With the site excluded, “Content from other sites will still be available in those features, and it may appear similar to yours.” And “This control doesn’t affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended.”
SEO Genome record, a client case investigation on August 2, 2026, internal file, client not identified. The robots.txt served by the main domain was collected with curl. The file had two parts. First, a block between the markers # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content, with Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot, and meta-externalagent. Then, the rules the team wrote in WordPress, with the comment referring to AI crawlers. OAI-SearchBot, PerplexityBot, and Claude-SearchBot were not on the list. The same block appeared on four of the same owner’s six domains; the other two had no AI block.
Cloudflare Docs, “robots.txt setting,” https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/, page last updated August 3, 2026. Accessed September 14, 2026. Passages: “When you turn on the managed robots.txt setting, Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content”; “Cloudflare will prepend our managed robots.txt before your existing robots.txt, combining both into a single response”; the example with the markers ”# BEGIN Cloudflare Managed content” and ”# END Cloudflare Managed Content”; the option “Set your preference to block training in robots.txt”; and “Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard. However, we have observed no impact on crawling rates or SEO as a result of these reports.”
Search Console Help, “robots.txt report,” https://support.google.com/webmasters/answer/6062598; Portuguese version, “Relatório robots.txt,” with an AI translation notice. Accessed September 14, 2026. Passages: “The robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered”; the columns for path, fetch status, checked on, size, and issues; the versions fetched “in the last 30 days”; the recrawl request, with “note that this doesn’t guarantee an immediate recrawl of unblocked URLs”; “This report is available only for properties at the domain level”; “Not having a robots.txt file is fine, and means that Google can crawl all URLs on your site”; and, for a specific address, “you can test the availability of the URL with the URL inspection tool”.

Robots.txt: What the File Controls, and What It Does Not Do