What is robots.txt
Robots.txt is a text file that allows the webmaster to disable or allow access to certain robots (bots), such as GoogleBot, SeznamBot and others.
The robots.txt module in the shop
The e-shop contains a module that allows you to extend the contents of the robots.txt file. The text box for adding robots.txt content is located in Settings ▸ Robots. No validation is applied to its content, so you can edit it without any restrictions.
You can use the Disallow, Allow and User-agent directives to determine which parts of the e-shop robots should or should not crawl. This allows you to make better use of your allocated Crawl Budget, i.e. to control which pages of your e-shop robots are allowed to visit within the time limit allocated for your website.
In the robots.txt file, you can also limit the frequency of crawling by robots using the Request-rate (List) and Crawl-delay (Bing, Yahoo,
Yandex, ...). By doing this, you can also reduce the total number of accesses by bots. Google restrictions can be set in the Google Search Console.
The robots.txt on the e-shop already contains this default block of directives:
User-agent: * - Defines a block of directives intended for all bots.
Sitemap - Link to sitemap.xml. So no need to list it in the administration.
Disallow: /admin/ - Disable browsing the administration.
Request-rate: 30/1m - Recommended crawl rate for ListBot. Maximum of 30 accesses per minute. In case you want to change it, it can be redefined by a directive in the block for a specific User-agent: SeznamBot. You can add this in the administration in the mentioned text field.
Crawl-delay: 10 - Recommended crawl speed (for Yahoo, Bing, Yandex and others). One access in 10 seconds. As for Request-rate, it can be redefined in the block for a specific User-agent.
You can read more about robots.txt in Google Help, Seznam.cz and the attached articles:
🇨🇿 support.google.com/webmasters/answer/6062608?hl=cs
🇨🇿 napoveda.seznam.cz/cz/fulltext-hledani-v-internetu/robots-txt
🇬🇧 www.robotstxt.org
🇬🇧 yoast.com/ultimate-guide-robots-txt
🇬🇧 www.contentkingapp.com/academy/robotstxt
🇬🇧 webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html