--- Title: How Googlebot indexes pages Language: en-US Canonical URL to this Markdown version of this page: [https://docs.kvik.shop/help/external-services/seo/how-googlebot-indexes-pages.md](https://docs.kvik.shop/help/external-services/seo/how-googlebot-indexes-pages.md) Canonical URL to HTML version of this page (for humans): [https://docs.kvik.shop/help/external-services/seo/how-googlebot-indexes-pages](https://docs.kvik.shop/help/external-services/seo/how-googlebot-indexes-pages) --- # How Googlebot indexes pages Breadcrumb: [Help](https://docs.kvik.shop/help.md) > [External services](https://docs.kvik.shop/help/external-services.md) > [SEO](https://docs.kvik.shop/help/external-services/seo.md) > [How Googlebot indexes pages](https://docs.kvik.shop/help/external-services/seo/how-googlebot-indexes-pages.md) An explanation of how Googlebot indexes pages. To effectively appear in search results, your page must be correctly indexed by Googlebot. When accessing your site, Googlebot sees a clearly understandable page structure, which it then indexes without any errors. GoogleBot then assigns a score to the indexed page - relevant to specific search phrases. Relevance is categorized by the content of the page as well as the overall strength of the site. The job of GoogleBot is to scan the site and index all the important pages. The bot enters the site with a certain amount of "time" for indexing. This time is well sufficient for regular sites with a few dozen pages. In the case of the e-shop with TecDoc integrated, however, there are millions of pages. For this reason, the average time is not enough to index the entire site. The bot bases the time spent on a page based on specific metrics. For example, if a page has a lot of high-quality backlinks directed to the site, it will be allocated with significantly more time. Likewise, if a bot finds out during indexing that it is interesting and unique content, it devotes more "time" to indexing. Unfortunately, in the case of TecDoc, the sites are very similar, reducing their attractiveness to the bot itself. As another example, pages with individual engines are practically identical, differing only in engine code and performance. The rest of the page is practically the same. There are many similar examples in the TecDoc database. The result is that the bot does not adequately prolong the "time" spent on the web due to its size. This pushes us to constantly work on innovations. One thing we've done is redesigned the structure for large parts of the site. This way, it’s clear to the bot that specific parts are not attractive content and don't need to be indexed. The Bot thus devotes his "time" to more important aspects. For example, product details. Unfortunately, these are also very similar on a massive scale. Of course, you can also improve the situation yourself. It is definitely beneficial to add unique and high-quality content directly to the web. This could, for example, mean writing useful articles for your visitors. This retains customers not only to buy more products but to learn more about the offered assortment and other non sales regarded information. You can also work on the link building. More about Google Search indexing. πŸ‡¬πŸ‡§ support.google.com/webmasters?hl=en#topic=9128571 πŸ‡¬πŸ‡§ developers.google.com/search/docs/guides/get-started How to restrict bot access. This can be done by using a robots.txt file. You can edit its content in Settings β–Έ SEO. There is also a robots.txt help there. By adjusting the crawl rate setting in the Google Search Console. This setting only affects GoogleBot. Googlebot Crawl Budget. πŸ‡¬πŸ‡§ webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html FAQ. Question: How can I bulk enable all important pages (products, categories, etc.) to be indexed? Answer: By default, all relevant pages are allowed to be indexed by Google Bot. Question: What happens after switching the shop from the onboarding URL to the production domain? Answer: After migrating the shop to the domain, robots start indexing your website. You can help robots find the shop using backlinks to the shop URL address. Robots might create a higher load in the first days after deployment of the shop to the domain. The scalable infrastructure is ready for that, but it may affect Pageviews package consumption. However, the number of consumed Pageviews should be covered by the basic package. You can track the pageviews usage in the admin interface -> Statistics agenda. Google Bot often crawls through the most interesting pages and avoids crawling all of the pages at the beginning. For reducing the unwanted crawling amount of Google Bot, please follow our Help page related to the Pageviews statistics. Question: Google Search Console is reporting that GoogleBot is getting a 444 return code. Is this OK? Answer: Yes, it is OK. GoogleBot will receive a return code of 444 if:. It is a GoogleBot user-agent, but the IP is not from Google. Is GoogleBot trying to index non-relevant pages such as search, shopping cart and user profile. GoogleBot accesses the API URL.