Hey guys,
I've had the need to easily prevent Google etc from crawling public testing sites (these are sites like staging.customerdomain.com that we show developments to customers on before the changes go live).
I've set up the module so that you can block the entire site, or you can block individual pages or folders using a checkbox in the CMS.
Check it out on GitHub: https://github.com/mobiusnz/silverstripe-noindex/
Let me know what you think!