Scan problems overview
You might experience scan issues with Morningscore's SEO checker. Here are the common reasons and possible solutions.
This guide and our proposed solutions require technical knowledge.
Problem: 401 Unauthorised
Visiting your website and receiving a 401 error code means that the web crawler is not authorized to access the requested resource. It typically happens when the web crawler fails to provide valid authentication credentials, such as a username and password.
In simple terms, it means that your website is protected by a password and the web crawler failed to provide the correct login information.
Solution: 401 Unauthorised
First of all, this might not be a problem. It's normal that some pages require a login and don't want crawlers to visit them. In that case, you can ignore this problem.
To fix this, you can allow a certain web crawler’s IP address(es) or user-agent to access specific parts of the website without the login, or provide the web crawler with a special login credentials—username and password—that it can use to access the website. We currently only support custom implementations in these scenarios, please contact us through the chat for possible offers.
Our user-agent is: "Morningscore Bot" (without "")
Example of how you allow our crawler in your robots.txt file:
For a list of our crawler IP's please reach out to us in the chat.
Problem: 403 Forbidden
A 403 response code means that the website is not allowing our web crawler to access its pages. The website may be blocking all web crawlers, or it may be blocking just certain ones. The website administrator may have set up rules to block web crawlers from accessing certain pages or sections of the website.
Solution: 403 Forbidden
You can fix a 403 error by allowing our web crawler to access your website. This can be done by:
adding the web crawler's IP address or user-agent to the website's whitelist which is a list of IP addresses or user-agents that are allowed to access the website.
Our user-agent is: "Morningscore Bot" (without "")
Example of how you allow our crawler in your robots.txt file:
For a list of our crawler IP's please reach out to us in the chat.
If you are using a security plugin, you need to check if that plugin is blocking the web crawler or not and adjust the settings accordingly.
You might be behind a strong firewall. It could be via a DNS provider like Cloudflare.
Your host / plugin on your website is blocking by IP list / ASN
All of our current Morningscore Bots are located in the ASN AS204287
So the fix in Cloudflare for example, would be to allow that ASN as shown here:
And then on that page fill this info in:
Problem: 451 Unavailable for legal reasons
When our web crawler visits your website and receives a 451 error code, it means that the web crawler is not able to access the requested resource due to legal reasons. The HTTP 451 error code is an HTTP status code specifically designed for situations when a website is blocked by legal authority.
In simple terms, it means that your website is blocked by the government or some legal authority and the web crawler is not able to access it.
Solution: 451 Unavailable for legal reasons
As a website owner, you will not be able to fix this problem by yourself, as the block is enforced by a legal authority. You may want to check with the legal authorities or consult with a legal expert to understand the reason for the block and what actions you can take to lift the block.
This is a rare problem.
Please contact us in the chat if you have this problem.
Problem: 429 Too Many Requests
When our web crawler visits your website and receives a 429 error code, it means you block us because we made too many "requests" to your website in a given amount of time. This is related to "rate limiting" the traffic.
In simple terms, it means your server blocks us because we scan too fast. In reality we do not scan that fast compared to similar tools, so it seems your server is very strict here.
Solution: 429 Too Many Requests
We are working on a solution to this problem on our end. Basically we will scan your website slower once we get this error code.
But you can certainly fix this on your end too. Whoever controls your server needs to reduce the "rate limiting". You can let them know about this specific problem, and they should know what to do.
Please contact us in the chat if you are stuck.
Problem: Robots.txt rules
When our web crawler visits your website and is blocked because of Robots.txt rules, it means that you have set up specific instructions for web crawlers in a file called "robots.txt" which is located on your website's root directory. This file contains rules that specify which pages or sections of your website should not be crawled or indexed by web crawlers.
In simple terms, it means that you have told web crawlers not to visit certain pages or sections of your website by adding them to the Robots.txt file.
Solution: Robots.txt rules
you can fix this problem by either removing the disallow directive from the Robots.txt file or by adding our web crawler's IP address or user-agent to the website's whitelist which is a list of IP addresses or user-agents that are allowed to access the website.
Our user-agent is: "Morningscore Bot" (without "")
Example of how you allow our crawler in your robots.txt file:
For a list of our crawler IP's please reach out to us in the chat.
Problem: Invalid Domain
When our web crawler tries to visit your domain, but for one reason or another is unable to access it, you'll see this error being reported.
Solution: Invalid Domain
Start out by visiting your domain in your browser, to check if the site can be accessed at all. If it cannot be accessed, you will need to either contact the administrator of your website, or troubleshoot the server yourself.
If you are able to access the site, try running a new scan as this may have been a temporary error. In case you are still receiving this error, please reach out to us in the chat with a screenshot of the error message.
This guide and our proposed solutions require technical knowledge.
Problem: 401 Unauthorised
Visiting your website and receiving a 401 error code means that the web crawler is not authorized to access the requested resource. It typically happens when the web crawler fails to provide valid authentication credentials, such as a username and password.
In simple terms, it means that your website is protected by a password and the web crawler failed to provide the correct login information.
Solution: 401 Unauthorised
First of all, this might not be a problem. It's normal that some pages require a login and don't want crawlers to visit them. In that case, you can ignore this problem.
To fix this, you can allow a certain web crawler’s IP address(es) or user-agent to access specific parts of the website without the login, or provide the web crawler with a special login credentials—username and password—that it can use to access the website. We currently only support custom implementations in these scenarios, please contact us through the chat for possible offers.
Our user-agent is: "Morningscore Bot" (without "")
Example of how you allow our crawler in your robots.txt file:
User-agent: Morningscore Bot
Allow: /For a list of our crawler IP's please reach out to us in the chat.
Problem: 403 Forbidden
A 403 response code means that the website is not allowing our web crawler to access its pages. The website may be blocking all web crawlers, or it may be blocking just certain ones. The website administrator may have set up rules to block web crawlers from accessing certain pages or sections of the website.
Solution: 403 Forbidden
You can fix a 403 error by allowing our web crawler to access your website. This can be done by:
adding the web crawler's IP address or user-agent to the website's whitelist which is a list of IP addresses or user-agents that are allowed to access the website.
Our user-agent is: "Morningscore Bot" (without "")
Example of how you allow our crawler in your robots.txt file:
User-agent: Morningscore Bot
Allow: /For a list of our crawler IP's please reach out to us in the chat.
If you are using a security plugin, you need to check if that plugin is blocking the web crawler or not and adjust the settings accordingly.
You might be behind a strong firewall. It could be via a DNS provider like Cloudflare.
Your host / plugin on your website is blocking by IP list / ASN
All of our current Morningscore Bots are located in the ASN AS204287
So the fix in Cloudflare for example, would be to allow that ASN as shown here:
And then on that page fill this info in:
Problem: 451 Unavailable for legal reasons
When our web crawler visits your website and receives a 451 error code, it means that the web crawler is not able to access the requested resource due to legal reasons. The HTTP 451 error code is an HTTP status code specifically designed for situations when a website is blocked by legal authority.
In simple terms, it means that your website is blocked by the government or some legal authority and the web crawler is not able to access it.
Solution: 451 Unavailable for legal reasons
As a website owner, you will not be able to fix this problem by yourself, as the block is enforced by a legal authority. You may want to check with the legal authorities or consult with a legal expert to understand the reason for the block and what actions you can take to lift the block.
This is a rare problem.
Please contact us in the chat if you have this problem.
Problem: 429 Too Many Requests
When our web crawler visits your website and receives a 429 error code, it means you block us because we made too many "requests" to your website in a given amount of time. This is related to "rate limiting" the traffic.
In simple terms, it means your server blocks us because we scan too fast. In reality we do not scan that fast compared to similar tools, so it seems your server is very strict here.
Solution: 429 Too Many Requests
We are working on a solution to this problem on our end. Basically we will scan your website slower once we get this error code.
But you can certainly fix this on your end too. Whoever controls your server needs to reduce the "rate limiting". You can let them know about this specific problem, and they should know what to do.
Please contact us in the chat if you are stuck.
Problem: Robots.txt rules
When our web crawler visits your website and is blocked because of Robots.txt rules, it means that you have set up specific instructions for web crawlers in a file called "robots.txt" which is located on your website's root directory. This file contains rules that specify which pages or sections of your website should not be crawled or indexed by web crawlers.
In simple terms, it means that you have told web crawlers not to visit certain pages or sections of your website by adding them to the Robots.txt file.
Solution: Robots.txt rules
you can fix this problem by either removing the disallow directive from the Robots.txt file or by adding our web crawler's IP address or user-agent to the website's whitelist which is a list of IP addresses or user-agents that are allowed to access the website.
Our user-agent is: "Morningscore Bot" (without "")
Example of how you allow our crawler in your robots.txt file:
User-agent: Morningscore Bot
Allow: /For a list of our crawler IP's please reach out to us in the chat.
Problem: Invalid Domain
When our web crawler tries to visit your domain, but for one reason or another is unable to access it, you'll see this error being reported.
Solution: Invalid Domain
Start out by visiting your domain in your browser, to check if the site can be accessed at all. If it cannot be accessed, you will need to either contact the administrator of your website, or troubleshoot the server yourself.
If you are able to access the site, try running a new scan as this may have been a temporary error. In case you are still receiving this error, please reach out to us in the chat with a screenshot of the error message.
Updated on: 04/01/2024