TL;DR: I can change MIDI instruments in my iOS app when running in the Xcode simulator, but not when running on-device.
I've been following a few of Gene De Lisa's posts, to learn how to use MIDI in an iOS app. I've gotten the furthest by following Multi-timbral AVAudioUnitMIDIInstrument to subclass AVAudioUnitMIDIInstrument
, and use the kMusicDeviceProperty_SoundBankURL
property to point at an sf2 sound font file.
In the Xcode simulator, this works great. I can sendProgramChange
to switch to any of the standard MIDI instrument. But, when I load my app onto an iPhone, only the piano, MIDI instrument 1, plays any sound. Has anyone else seen this behavior?
The code I'm using is based on the "iOS Game" template in Xcode 11.4, with four changes.
The first change is to setup the audio session for the app properly. My new AppDelegate.application
:
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
let audioSession = AVAudioSession.sharedInstance()
do {
try audioSession.setCategory(.playback, options: .duckOthers)
} catch let error as NSError {
print("Unable to set audio category: \(error.localizedDescription)")
}
do {
try audioSession.setActive(true)
} catch let error as NSError {
print("Unable to activate audio session: \(error.localizedDescription)")
}
return true
}
The second is my AVAudioUnitMIDIInstrument
subclass:
class MyMIDIInstrument: AVAudioUnitMIDIInstrument {
init(soundBankURL: URL) throws {
let description = AudioComponentDescription(
componentType: kAudioUnitType_MusicDevice, componentSubType: kAudioUnitSubType_MIDISynth, componentManufacturer: kAudioUnitManufacturer_Apple, componentFlags: 0, componentFlagsMask: 0
)
super.init(audioComponentDescription: description)
var bankURL = soundBankURL
let status = AudioUnitSetProperty(
self.audioUnit,
AudioUnitPropertyID(kMusicDeviceProperty_SoundBankURL),
AudioUnitScope(kAudioUnitScope_Global),
0,
&bankURL,
UInt32(MemoryLayout<URL>.size))
if (status != OSStatus(noErr)) {
throw NSError(domain: "MyMIDIInstrument", code: Int(status), userInfo: [NSLocalizedDescriptionKey: "Could not set soundbank property"])
}
}
}
The final code change is adding initialization and play in GameScene
:
class GameScene: SKScene {
var avEngine = AVAudioEngine()
var midi: MyMIDIInstrument?
override func didMove(to view: SKView) {
guard let soundBankURL = Bundle.main.url(forResource: "FluidR3_GM", withExtension: "sf2")
else {
print("Failed to get url for sound bank")
exit(-1)
}
do {
try midi = MyMIDIInstrument(soundBankURL: soundBankURL)
} catch let error as NSError {
print("Failed to init MIDI: \(error.code) - \(error.localizedDescription)")
exit(-1)
}
avEngine.attach(midi!)
avEngine.connect(midi!, to: avEngine.mainMixerNode, format: nil)
do {
try avEngine.start()
} catch let error as NSError {
print("Failed to start AVEngine: \(error.localizedDescription)")
}
// Case 1: do not send program change
//
// Results:
// Simulator: piano sounds
// iPhone: piano sounds
// Case 2: send program change to 0=piano, without bank
// midi!.sendProgramChange(UInt8(0), onChannel: UInt8(0))
// Results:
// Simulator: piano sounds
// iPhone: piano sounds
// Case 3: send program change to 0=piano, with bank=melodic
// midi!.sendProgramChange(UInt8(0), bankMSB: UInt8(kAUSampler_DefaultMelodicBankMSB), bankLSB: UInt8(kAUSampler_DefaultBankLSB), onChannel: UInt8(0))
// Results:
// Simulator: no sound
// iPhone: no sound
// Case 4: send program change to 0=piano, with bank=0
// midi!.sendProgramChange(UInt8(0), bankMSB: UInt8(0), bankLSB: UInt8(kAUSampler_DefaultBankLSB), onChannel: UInt8(0))
// Results:
// Simulator: piano sounds
// iPhone: piano sounds
// Case 5: send program change to 12=marimba, without bank
// midi!.sendProgramChange(UInt8(12), onChannel: UInt8(0))
// Results:
// Simulator: marimba sounds
// iPhone: no sound
// Case 6: send program change to 12=marimba, with bank=melodic
// midi!.sendProgramChange(UInt8(12), bankMSB: UInt8(kAUSampler_DefaultMelodicBankMSB), bankLSB: UInt8(kAUSampler_DefaultBankLSB), onChannel: UInt8(0))
// Results:
// Simulator: no sound
// iPhone: no sound
// Case 7: send program change to 12=marimba, with bank=0
// midi!.sendProgramChange(UInt8(12), bankMSB: UInt8(0), bankLSB: UInt8(kAUSampler_DefaultBankLSB), onChannel: UInt8(0))
// Results:
// Simulator: marimba sounds
// iPhone: no sound
}
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
print("playing note")
midi!.startNote(64, withVelocity: 64, onChannel: 0)
}
}
And the final change is to add the "FluidR3_GM.sf2" sound font to the project.
In GameScene.didMove(toView:)
, you can see the various behaviors I've seen. I find a few things interesting:
I can get piano sounds both in the simulator and on-device, in any of three ways: by never calling
sendProgramChange
, by callingsendProgramChange
without specifying a bank MSB/LSB, or by callingsendProgramChange
with MSB set to zero. However, if I callsendProgramChange
specifying MSB askAUSampler_DefaultMelodicBankMSB
, as all instructions I've seen say you should, I get no sound.I can get marimba sounds out of the simulator by calling
sendProgramChange
in any of the ways that worked for getting piano sounds. But, none of these methods work on-device.
I have tried the same experiment with the GeneralUser GS sound font, with the same results.
So, does anyone have an idea of what I may be missing?