1) I you don't mind sacrificing size of file, then you can try the fastest preset:
-preset ultrafast
2) Also be sure you're using all cores (I've used linux ffmpeg builds that didn't seem to do this by default) by setting:
-threads 0
Here's your final command, setting these parameters for each downscale:
exec('ffmpeg -i 1080p.mp4 -threads 0 -preset ultrafast -s 800x480 -c:v libx264 480p.mp4 -threads 0 -preset ultrafast -s 640x360 -c:v libx264 360p.mp4 -threads 0 -preset ultrafast -s 320x240 -c:v libx264 240p.mp4');
The above command doubled my speed, but more than doubled the file sizes.
3) Short of this, you might want to consider adding a higher/faster core CPU. Use the benchmarks here to find the best price/speed ratio.
4) You could also consider building ffmpeg with QuickSync support if you have an intel CPU that supports it. Alteratively you could add an NVIDA video card and do the encoding on the card via NVENC. Details for hardware encoding are here.