This has been asked in the past with no conclusive answer.
What would be sensible content? Can anyone up' an example?
Much appreciated.
This site requires you to update your browser. Your browsing experience maybe affected by not having the most up to date version.
Please use forum.silverstripe.org for any new questions
(announcement).
The forum archive will stick around, but will be read only.
You can also use our Slack channel
or StackOverflow to ask for help.
Check out our community overview for more options to contribute.
This has been asked in the past with no conclusive answer.
What would be sensible content? Can anyone up' an example?
Much appreciated.
Hi,
The following links may help:
http://www.robotstxt.org/
http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
Ben
Generally, they do. So thank you.
But I was seeking a SS-specific example / recommendations.
Is there something specific you're trying to accomplish? Generally, I think a normal SS install should be fine without a robots.txt file unless you wanted to do something like http://silverstripe.com/robots.txt.
Perhaps we should bundle a default robots.txt file with the installer?
User-agent: *
Disallow: /admin
Disallow: /assets
Sounds like a good idea, Sam.
Not sure if assets/ should be included in the exclusion list, though. I can see some people liking it there because they don't want their assets showing up in search engines separately from the pages holding those assets. On the other hand, there are those who want their assets to be indexed so that they will show up in things like Google image search. The "assets/" directive would prevent that.
Ben
Disallowing assets doesn't make sense to me since there might be pdf or txt files which are useful content for search engines.
It would make sense to disallow cache flushing by search engines:
User-agent: *
Disallow: /admin
Disallow: /?flush
I use
User-agent: *
Disallow: /admin
Disallow: /?flush
Disallow: /myotherprivatedirectory
Allow: /
User-Agent: Googlebot-Image
Disallow: /admin
Disallow: /?flush
Disallow: /myotherprivatedirectory
Disallow: /assets
Allow: /