How to scan specific pages only in Health
If you have a website that covers several countries or industries, and only want to see the health data for one of them, there is a way to do that with Morningscore:
Change your robots.txt
To only allow Morningscore's SEO checker (bot that crawls in technical terms) to scan certain pages, you can set up rules for that in your robots.txt.
Follow this guide:
First check how your robots.txt looks like by going to yourwebsite.com/robots.txt (add your own website)
Usually it would look something like this (WordPress example):
This is the code you should add to it - example:
In this example you say that all pages with /de/ in the URL can be scanned by Morningscore, but otherwise no other pages. This would be the example of Morningscore only scanning German pages on your global website (if /de/ is used for German pages).
Replace the above code example with your situation
If you for example only want to scan your shop and all URL's contain /shop/ then this is the code you would use:
Add code to the robots.txt AFTER the code already there.
Normally, what you want to do, is add this new code to the robots.txt AFTER the code already there.
Here is an example:
The code In the red box is the new code. And as you can see, it is put AFTER the current code that we found in step 2.
Fully customize pages to scan
You can even add more URL paths by simply following this example:
So by adding more Allow: entries you can fully customize which URL paths we should crawl.
How do I change the robots.txt code?
Some CMS like WordPress can do it via plugins (often SEO plugins like Yoast). In other cases you need to change the robots.txt manually.
You need to talk to your technical contact if you are in doubt.
If you are not able or allowed to change your robots.txt we also have a solution for an XML file morningscore_whitelist.xml. This file needs to be uploaded to your websites root directory. If you think this could be a better option, talk to us on the chat and we will guide you
This will not affect Google
This has no implication for Google's robots or anything else, since you are saying this only counts for Morningscore's robot with this line:
User-agent: Morningscore Bot
Don't forget that line!
If you need any help with this, you are very welcome to reach out to us on the chat
Change your robots.txt
To only allow Morningscore's SEO checker (bot that crawls in technical terms) to scan certain pages, you can set up rules for that in your robots.txt.
Follow this guide:
First check how your robots.txt looks like by going to yourwebsite.com/robots.txt (add your own website)
Usually it would look something like this (WordPress example):
This is the code you should add to it - example:
User-agent: Morningscore Bot
Allow: /de/
Disallow: /In this example you say that all pages with /de/ in the URL can be scanned by Morningscore, but otherwise no other pages. This would be the example of Morningscore only scanning German pages on your global website (if /de/ is used for German pages).
Replace the above code example with your situation
If you for example only want to scan your shop and all URL's contain /shop/ then this is the code you would use:
User-agent: Morningscore Bot
Allow: /shop/
Disallow: /Add code to the robots.txt AFTER the code already there.
Normally, what you want to do, is add this new code to the robots.txt AFTER the code already there.
Here is an example:
The code In the red box is the new code. And as you can see, it is put AFTER the current code that we found in step 2.
Fully customize pages to scan
You can even add more URL paths by simply following this example:
User-agent: Morningscore Bot
Allow: /shop/
Allow: /blog/
Allow: /help/
Disallow: /So by adding more Allow: entries you can fully customize which URL paths we should crawl.
How do I change the robots.txt code?
Some CMS like WordPress can do it via plugins (often SEO plugins like Yoast). In other cases you need to change the robots.txt manually.
You need to talk to your technical contact if you are in doubt.
If you are not able or allowed to change your robots.txt we also have a solution for an XML file morningscore_whitelist.xml. This file needs to be uploaded to your websites root directory. If you think this could be a better option, talk to us on the chat and we will guide you
This will not affect Google
This has no implication for Google's robots or anything else, since you are saying this only counts for Morningscore's robot with this line:
User-agent: Morningscore Bot
Don't forget that line!
If you need any help with this, you are very welcome to reach out to us on the chat
Updated on: 30/01/2024