Does anyone have a solution for getting the hits from Googlebot? It's creating a ton (literally 1476) session files on my server (causing them to be very unhappy) all with zero byte count. I tried blocking the IP and they come in on another one. They are hitting twice a minute . . . they keep trying my old calendar install. When they get the 404 Silverstripe is showing the "page missing" message and I suspect Google is going "ok, and we'll try back again since you didn't send a 404 message". Any ideas?
We've moved the forum!
Please use forum.silverstripe.org for any new questions
(announcement).
The forum archive will stick around, but will be read only.
You can also use our Slack channel
or StackOverflow to ask for help.
Check out our community overview for more options to contribute.
Try a robots.txt...
Allow: /pages/spider.php
Crawl-delay: 10
User-agent: *
Disallow: /webcal/
Disallow: /calendar/
Nope . . . still hammering it
ok, you can tweak the robots file to tell various agents to ignore various parts of the site.
Maybe if you don't care for SEO just disallow all?
User-agent: *
Disallow: /