Sixty-Eight Percent of What Google Does on Your Site Is Rechecking What Is Already There

Whenever crawl budget comes up, the conversation tends to stop at “Google visits your site this many times a day”.

The part that matters is missing: what for. And there is a screen that splits exactly that into two columns.

The two questions Google answers when it crawls

The Crawl stats report has a block called By purpose. It splits everything in two, and both definitions are official.

Discovery: “the URL requested was never crawled by Google before”. ¹

Refresh: “a recrawl of a known page”. ²

Find new things, recheck old ones. Every crawl of your site fits in those two.

This was the split on the example site, on August 30, 2026.

Refresh     68%
Discovery   32%

³

Two thirds of the effort went to rechecking pages Google already knew.

That is normal for a site that has already been discovered. It turns into information in one specific situation. You published forty new pages and discovery did not rise. Then the new material is not being found, and the reason is usually internal links or the sitemap.

What crawl budget is, in the words of the people who coined the term

Google defines it this way: crawl budget is “the set of URLs that Google can and wants to crawl”. ⁴

Look at the two verbs. Can and wants. They are separate things, and Google names them.

The first one is capacity. The document calls it the crawl capacity limit, also known as hostload, and it caps “the total amount of time your server spends holding connections open”. If your server is slow or fails, Google eases off on its own. ⁵

The second one is appetite. The document calls it crawl demand, and it varies with site size, update frequency, quality, and relevance. ⁵

A fast server solves the first. It does not solve the second.

The warning Google itself gives, and almost nobody repeats

The crawl budget guide declares itself an “advanced guide intended primarily for” sites above 1 million pages with weekly changes, or above 10,000 pages with daily changes. ⁶

And it is explicit about who does not need it: “you don’t need to read this guide” if your pages are crawled on the same day you publish them. ⁷

I keep that here because crawl budget turned into a sales pitch. If your site has two hundred pages and they show up in Google the next day, you do not have a crawl budget problem. You have another one, and it is worth finding out which before buying a solution for this one.

Who is crawling, and why that matters

Another block on the same screen splits the requests by Googlebot type. In the example:

Smartphone            38%
Other agent type      28%
Image                 19%
Desktop               13%
AdsBot                 2%

⁸

Three readings come out of that.

Smartphone ahead of desktop, 38% against 13%. That is the expected behavior since mobile-first indexing. Seeing the number confirms your site is being read the way most sites are.

Image at 19%. Almost a fifth of the crawling going to image files. On a site with many heavy photos that number climbs, and it eats room that would go to content.

AdsBot at 2%, on a site that runs no ads. And here comes a piece of information that changes what you do with robots.txt.

The three crawler categories, and the rule for each one

Google splits its own crawlers into three groups. Each one behaves differently in front of robots.txt. ⁹

The AdsBot in the example belongs to the second category. Blocking it in robots.txt does not guarantee it stops.

One note on precision. The names shown in the Search Console panel are interface labels. They are not necessarily the technical names in the crawler documentation, and that is why I cite each one for what it is. ¹⁰

What to do with this on Monday

Open the screen and answer three questions.

What is your split between refresh and discovery? If you published a lot recently and discovery is low, the problem is discovery, not content.

What share goes to images? If it passes a quarter on a site that is not about photography, file weight is worth a look.

Does any agent show up that you were not expecting? AdsBot on a site with no ads, for one, is a fair question for whoever runs the infrastructure.

And before any of that, one question decides whether the rest matters. How fast do your new pages show up in Google? If they show up fast, Google already answered that you do not need to optimize this.

Frequently asked questions

What is crawl budget, in one sentence?

In the official definition, it is the set of URLs Google can and wants to crawl. Capacity and appetite, two separate things. ^{4 5}

Does my small site need to worry about this?

The official guide says no, if pages are crawled on the same day they are published. It is aimed at sites above 10,000 pages with daily changes, or above 1 million with weekly changes. ^{6 7}

Does blocking AdsBot in robots.txt work?

AdsBot is a special-case crawler, and the documentation states that this category may not obey robots.txt. ⁹

Does a faster server increase my crawling?

It can raise capacity, which is half the equation. The other half is demand, and it depends on size, update frequency, quality, and relevance. ⁵

Notes and sources

1. Discovery definition: Search Console Help, “Crawl Stats report”, support.google.com/webmasters/answer/9679690, consulted on September 1, 2026.
2. Refresh definition: same source as note 1.
3. Google Search Console, seodeentidade.com property, Crawl stats, “By purpose” block. Capture of September 1, 2026, report last updated on August 30, 2026. The property is the author’s own site, published in Portuguese.
4. Google Search Central, “Large site owner’s guide to managing your crawl budget”, developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget, consulted on September 1, 2026. Literal quotation reproduced in the body.
5. Components of the budget in the same source as note 4: crawl capacity limit, also called hostload, and crawl demand. The literal quotation on the capacity limit is reproduced in the body.
6. Declared audience of the guide, same source as note 4.
7. Statement on who does not need the guide, same source as note 4. Literal quotation reproduced in the body.
8. Google Search Console, same property and capture as note 3, “By Googlebot type” block.
9. Crawler categories: Google Search Central, crawlers and Googlebot verification documentation, developers.google.com/search/docs/crawling-indexing/verifying-googlebot, consulted on September 1, 2026. Common crawlers obey robots.txt, special-case crawlers may not obey it, and user-triggered fetchers ignore it.
10. Declared limitation: the labels displayed in the Search Console panel are not necessarily the technical names in the crawler documentation. The nominal list of agents did not open in the source consulted on September 1, 2026.