Is there a CMS function to do this without having to edit robots?

Did you install the GoogleSitemaps module? That should take care of that:

http://addons.silverstripe.org/add-ons/silverstripe/googlesitemaps

Thanks for the reply, but how does this sitemap module set a robotos noindex tag?

No, it creates a sitemap.xml that Google and other engines use to index pages. You can add priorities 1 to 10, and noindex.

OK thanks. But Google may not use the sitemap file as an entry point to the page, nor does it translate robots directives from a sitemap file. The robots directive must be either in the root as a robots.txt file - which controls crawl to folders of files, or via the page html head to request noindex. Unless you can teach me something new about this topic....

I see the info about this module states:

"SilverStripe support for the Google Sitemaps XML, enabling Google and other search engines to see all urls on your site. This helps your SilverStripe website rank well in search engines, and to encourage the information on your site to be discovered quickly."

This is somewhat misleading as sitemaps have no effect on ranking at all. Furthermore, the sitemap is an unlinked file and is not discovered unless submitted to search engines. Crawl rate is determined by crawl budget and the amount of external referral, so the sitemap alone doesn't cause crawl, rank, or page priority in search results.

OK, well, I can't really comment on that as I'm no SEO expert, I just do the programming :)

There is this module http://addons.silverstripe.org/add-ons/nobrainerweb/silverstripe-robots-noindex but it looks as if that one will only change robots.txt if you're in dev mode so nothing gets indexed...

Shouldn't be too hard to add a textfield to SiteConfig settings, containing the robots text, and updating from there - onAfterWrite(). And if you want page-specific settings it would be a bit more work, but not really complex...

But I think

Thanks Martine.
Yes, at the moment we have decided to address this with writing our own code for it. I just thought that because it was such a common requirement that a module or native function might already exist. I'm no coder. Maybe this is harder to build than I realise.