Right - tuning the webserver (Apache) is the place where you'll get some of the least performance gains.
You have extra resources available, so you're on the right track - you need to put them to use. It's time to use your memory much more aggressively to speed up your side - this is where you're going to get the most performance gains. Reduce CPU usage, increase memory usage (through caching), minimise disk access.
You don't mention what OS you're running on, or what programming language you're using. My area of expertise is Linux and PHP, so I'll go with some advice for that and if its wrong you might be able to adapt it for other stacks.
PHP - are you using something like APC or xCache? If not, get on it! Make sure you allocate enough memory in the config to contain your whole application.
Start building in caching into your application - make use of things like memcached (you can also probably use xcache/APC variable storage).
Do some analysis of the database usage. Increase the buffers and caches to keep more stuff in memory => less disk access.
Many people concentrate on minimising your memory usage, but what use is a server with 8gbs of RAM if you're only using 3. Obviously minimise any memory usage that is unncessary or not actually serving a purpose, but other than that, use your resources!