What is a crawl delay in my robots.txt?
Maybe Morningscore's health scanner warned you that we crawl slow because of a crawl delay rule in your robots.txt and you ended up here.
Or you found this guide by searching.
In either case, let us try to explain Crawl-delay in robots.txt
Check your own robots.txt
First go to yourwebsite.com/robots.txt (replace first part with your own website).
Now you should see a text file in your browser.
Try to search that for "Crawl-delay" in the text (Ctrl+F). If something is found, it probably looks similar to this:
If crawl delay is 10 then that means we wait 10 seconds after each page scan has finished. So 10 as the example above means max 6 pages on your website is crawled per minute by web crawler robots like Morningscore and Google.
Change Crawl-delay for Morningscore only
You can make sure that Morningscores scans at normal speed even if you generally have Crawl-delay enabled for other robots like this:
So here you say that Morningscore's robot can crawl at our normal speed with "Crawl-delay: 0". You basically exclude us from the rule. This whole part should be put at the bottom of your robots.txt file for it to work:
You could also set it to Crawl-delay: 3, 10 or even 60 (60 is our max respected value - which is 60 secs or 1 min delay per page we scan)
Test if it works by doing another full scan with Morningscores health scanner and observing that we scan faster + there is no crawl delay warning anymore.
Is it smart to use Crawl-delay?
Crawl-delay is not something we really recommend, since you slow robots like Google and Morningscore down. But there can be legit reasons for it, that you should ask your technical contact for. If your server can't handle faster speeds, then it's a very weak server generally speaking, and that should in our opinion question whether that's a wise choice. More server power is cheap in todays competitive hosting world.
Or you found this guide by searching.
In either case, let us try to explain Crawl-delay in robots.txt
Check your own robots.txt
First go to yourwebsite.com/robots.txt (replace first part with your own website).
Now you should see a text file in your browser.
Try to search that for "Crawl-delay" in the text (Ctrl+F). If something is found, it probably looks similar to this:
User-agent: *
Crawl-delay: 10If crawl delay is 10 then that means we wait 10 seconds after each page scan has finished. So 10 as the example above means max 6 pages on your website is crawled per minute by web crawler robots like Morningscore and Google.
Change Crawl-delay for Morningscore only
You can make sure that Morningscores scans at normal speed even if you generally have Crawl-delay enabled for other robots like this:
User-agent: *
Crawl-delay: 60
User-agent: Morningscore Bot
Crawl-delay: 0So here you say that Morningscore's robot can crawl at our normal speed with "Crawl-delay: 0". You basically exclude us from the rule. This whole part should be put at the bottom of your robots.txt file for it to work:
User-agent: Morningscore Bot
Crawl-delay: 0You could also set it to Crawl-delay: 3, 10 or even 60 (60 is our max respected value - which is 60 secs or 1 min delay per page we scan)
Test if it works by doing another full scan with Morningscores health scanner and observing that we scan faster + there is no crawl delay warning anymore.
Is it smart to use Crawl-delay?
Crawl-delay is not something we really recommend, since you slow robots like Google and Morningscore down. But there can be legit reasons for it, that you should ask your technical contact for. If your server can't handle faster speeds, then it's a very weak server generally speaking, and that should in our opinion question whether that's a wise choice. More server power is cheap in todays competitive hosting world.
Updated on: 29/01/2024