I always include this in my .htaccess file:
### Performance Tweaks ###
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType text/html "access plus 30 minutes"
ExpiresByType image/gif "access plus 30 day"
ExpiresByType image/png "access plus 30 day"
ExpiresByType image/jpg "access plus 30 day"
ExpiresByType image/jpeg "access plus 30 day"
ExpiresByType image/ico "access plus 1 year"
ExpiresByType text/css "access plus 14 day"
ExpiresByType text/javascript "access plus 14 day"
ExpiresByType application/x-javascript "access plus 14 day"
</IfModule>
<IfModule mod_headers.c>
Header unset ETag
RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match
FileETag None
Header set Connection keep-alive
</IfModule>
<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images, movies or zip files
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:avi|mov|mp3|mp4|rm|flv|swf|mp?g)$ no-gzip dont-vary
<IfModule mod_headers.c>
Header append Vary User-Agent
</IfModule>
</IfModule>
### End Performance Tweaks ###
Another thing that might be worth looking into is partial caching:
http://doc.silverstripe.org/framework/en/reference/partial-caching
Optionally:
I would look into an affordable cdn for images, videos, css and javascript for better download speeds on those kind of files
Finally:
You always need to step back a little bit and think about what's going on the server. If you have a lot of logic it's quite expected to be slowish so in that case you might consider a stronger host or a hosting service which is located closer to your target audience. I suggest a VPS