I'd like to add an extra class to a gridfield row (the tr element) is there a (standard) way of doing this or do I have to write my own hack? I couldn't find anything in the API, but I could have missed something. Tried asking on IRC but didn't get a response.

Unfortunately not thus far - http://api.silverstripe.org/3.1/source-class-GridField.html#473
Only columns it seems - http://api.silverstripe.org/3.1/class-GridFieldDataColumns.html

Perhaps you could affix a class to the first column or something?
Maybe through a custom component implementing http://api.silverstripe.org/3.1/class-GridField_HTMLProvider.html

I hadn't noticed the refractored GridField class from April 17/20. Hack was relatively simple with new code:

RowExtraClassesGridField.php

<?php

class RowExtraClassesGridField extends GridField {
//  class RowExtraClassesGridField extends FrontEndGridField {
	protected function newRowClasses($total, $index, $record) {
		$classes = array('ss-gridfield-item');

		if($index == 0) {
			$classes[] = 'first';
		}

		if($index == $total - 1) {
			$classes[] = 'last';
		}

		$classes[] = ($index % 2) ? 'even' : 'odd';

		if ($record->hasMethod("GridFieldRowClasses")) $classes = array_merge($classes, $record->GridFieldRowClasses());

		return $classes;
	}
}

Rather than create GridField create RowExtraClassesGridField

		$gridField = new RowExtraClassesGridField(
			$this,
			"Message",
			$data,
			$config
		);

Add "GridFieldRowClasses" function to your DataObject:

	public function GridFieldRowClasses() {
		$classes = array();
		if ($this->UnreadMessages()) $classes[] = "unread-messages";
		return $classes;
	}

Unfortunately using a DataExtension didn't seem to work, which would have been a nicer method.

// RowExtraClassesGridField.php
class RowExtraClassesGridField extends DataExtension {
  ::
-->8--
//config.yml

GridField:
  extensions:
    - RowExtraClassesGridField

Your class can be further condensed:

class RowExtraClassesGridField extends GridField {

	protected function newRowClasses($total, $index, $record) {
		$classes = parent::newRowClasses($total, $item, $record);
		if($record->hasMethod('GridFieldRowClasses'))
			$classes = array_merge($classes, $record->GridFieldRowClasses());
		return $classes;
	}
}

Instead of DataExtension you can always leverage the injector to pick up your class, e.g. by adding this to your YAML file:

Injector:
    GridField:
        class: RowExtraClassesGridField

In this case, I suspect you should instantiate your grid fields with GridField::create() instead of new GridField() though.