Should any of the following lines be included in robots.txt?
Disallow: /cms/
Disallow: /sapphire/
Disallow: /mysite/
Disallow: /userforms/
Disallow: /themes/
Please help! Thanks.
This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.
Please use forum.silverstripe.org for any new questions
(announcement).
The forum archive will stick around, but will be read only.
You can also use our Slack channel
or StackOverflow to ask for help.
Check out our community overview for more options to contribute.
Should any of the following lines be included in robots.txt?
Disallow: /cms/
Disallow: /sapphire/
Disallow: /mysite/
Disallow: /userforms/
Disallow: /themes/
Please help! Thanks.
If it is still relevant... :)
The CMS automatically redirects the requests you are disallowing.
my robots.txt looks like this
==============================
User-agent: *
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Disallow: /assets
Allow: /
User-agent: ia_archiver
Disallow: /
User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /dev
Disallow: /?flush
Allow: /
Sitemap: http://www.mypage.com/sitemap.xml
==============================