In early 2024, Google ended support for its crawl reduction request tool from the original version of Google Search Console. Google also changed its crawling strategies towards the end of 2023 and especially at the beginning of 2024.
According to Google's help on this topic, it is now recommended to regulate access on the website with return codes. For example, code 429 "Too many requests".
Due to the significant increase in page views by Googlebot, on 10.1.2024 we started so-called throttling, where we are currently handling part of the requests of this robot with the return code 429. The default values for this throttling are set so that Google does not exhaust your basic infrastructure package. So far, this package has been sufficient even for Google, and it normally used up to 20% of it.
Googlebot crawls several services at once under the same agent ID.
If you want to increase the throughput of Googlebot requests, you can do that in Access blocking interface.
FAQ
Question: Google says the block should be short-lived, but we see the block all day long.
Answer: Blocking occurs at a granularity of one minute. It is therefore even shorter than short-term, rather momentary, a few tens of seconds at most.
Question: We are concerned that throttling may affect the sales performance of the e-shop. How to solve this?
Answer: Practically any change to the e-shop can affect sales performance. The biggest factor is seasonality and customers shopping mood. For example, there were big changes in 9/2023 and 12/2023.
Throttling can affect page indexing. Here, we recommend evaluating the possible impact not in the form of page indexing (Google regularly excludes non-converting pages from the index), but of sales metrics, for example, in Google Analytics (be careful, during the evaluation, you should exclude sales realized through the searched brand words of your brand).
Furthermore, Throttling can affect product advertising from the feed in Google Merchant Center. Here you need to turn off the additional check of products on the site and request the activation of rejected items.
For both channels, temporarily increasing the throttling bandwidth limits can help.
Question: Google mentions unavailability with a 5xx return code in their services, but you mention throttling with a 4xx return code. How is it possible?
Answer: Throttling does not occur using 5xx. It only uses the return code 429. The mention of 5xx may be an error on Google's part, or a historically recorded condition when the URL was previously unavailable.
Question: We have a large number of rejected products in Google Merchant Center (GMC) due to the unavailability of URLs. How can we solve the situation?
Answer: Check with GMC that you are not allowed to post-check/update products from the website. This substantially increases the crawl rate and causes other errors such as price mismatch. You should leave this additional check/update turned off.
Also, request an inspection of the affected products. You can increase the throughput of Googlebot requests during the review period, but keep in mind that Google can generate a really high volume of traffic.
Question: Google warns us that the product landing page is unavailable. However, at the time of our inspection, the certificate is available. How is it possible?
Answer: When throttling, we distinguish between traffic and it only applies to Googlebot traffic. It is not possible for you to access the site as a Googlebot.
Question: When accessing a website with a Googlebot user-agent, the website gives us a return value of 444.
Answer: If you are accessing a website with Googlebot identification, but you are not accessing it from the IP address that Google provides for Googlebot, then these accesses are rejected with a return code of 444.
Question: We send site URLs to Google via sitemap.xml but it doesn't load all of them. How to make it load all?
Answer: It is not possible to ensure that Google loads all pages. The sitemap informs him about the pages listed in sitemap.xml. Google CAN load them (Google does not guarantee loading). If it is paused by throttling while loading, then it can load the page on a repeat request later.
Question: Throttling is set, but the number of Googlebot hits does not change. How is it possible?
Answer: We know from real measurements that Google's response takes weeks. it is common that even after setting throttling, the number of accesses continues to grow for more than 7 days. The subsequent decline is gradual and may also last several weeks.