AFAIK you can't do this easily in .htaccess. I spent a while banging my head on it. There are solutions if you want to hardcode your base url but this does not work for the framework so not suitable.
See http://open.silverstripe.org/ticket/6700 for a bit of a discussion on this as well as http://pastie.org/2373435. It doesn't trim the extension off like it probably should (didn't notice that behaviour).