Ok have found something that works. Solves both the alarm issue and allows me to play the voicemail through the earpiece.
I needed to setMode back to what it was when I started, in onStop():
// set mode back to normal, and speakerphone back to true.
AudioManager am = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE);
am.setMode(prevMode);
am.setSpeakerphoneOn(true);
and in init:
private void initMediaPlayer() {
// Make the media play through earpiece instead of through speaker(default)
AudioManager am = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE);
prevMode = am.getMode();
am.setMode(AudioManager.MODE_IN_COMMUNICATION);
am.setSpeakerphoneOn(false);
Thanks @DreadfulWeather for leading me to this answer, with some helpful questions.