Turns out the Android Media Player (stagefright) has some weird buffering algorithm for variable bitrate files. It seems to try to buffer multiple megabytes (2-3 MB on a 5MB total MP3 file) until it starts playing. To make things worse, it may start the playback after buffering a large enough amount (e.g. 2MB), then stop playback again, wait until it buffers another MB before it resumes.
This happens only to VBR MP3 files to my experience and only on stagefright. So, is there a way to work around this?