Decoding Speex audio and playing decoded PCM data using AudioQueues

Question

I am getting audio stream data in Speex format. I tried compiling ffmpeg for Speex for Speex decoding. I am getting errors when compiling ffmpeg which can be seen in this link. So I used another option of using this framework for decoding Speex audio. I am playing the decoded data i.e. PCM data using Audio Queues. Following is the code for decoding Speex data and playing PCM data.

SpeexPlayer.h

#import <Foundation/Foundation.h>
#import <AVFoundation/AVFoundation.h>
#import <AudioToolbox/AudioToolbox.h>
#import "RtmpMediaStream.h"
#import "RtmpAVUtils.h"
#import <Speex/speex.h>
#import <Speex/speex_stereo.h>
#import <Speex/speex_callbacks.h>

typedef enum kAudioState {
    AUDIO_STATE_PAUSE,
    AUDIO_STATE_STOP,
    AUDIO_STATE_PLAYING,
    AUDIO_STATE_READY
} kAudioState;
typedef SInt16 AudioSampleType;
#define kNumAQBufs 3
#define kAudioBufferSeconds 3
@interface SpeexPlayer : RtmpAVUtils<RtmpMediaStreamAudioDataDelegate>{

    SpeexBits bits;
    void *dec_state;
    BOOL started;
    int frame_size;
    kAudioState state;
    AudioQueueRef audioQueue;
    AudioQueueBufferRef audioQueueBuffer;

}
- (id)initWithStream:(RtmpMediaStream *)stream;
- (void)audioQueueOutputCallback:(AudioQueueRef)inAQ inBuffer:(AudioQueueBufferRef)inBuffer;
- (void)audioQueueIsRunningCallback;
@end

SpeexPlayer.m

#import "SpeexPlayer.h"
#import "RtmpConstants.h"
#import "RtmpMediaStream.h"
#import "RtmpPacket.h"
int sample_rate = 11025;
int _lastTimestamp = 0;
BOOL isStarted = NO;
AudioTimeStamp tempTimeStamp;

void audioQueueOutputCallback(void *inClientData, AudioQueueRef inAQ,
                              AudioQueueBufferRef inBuffer) {
    SpeexPlayer *audioController = (__bridge SpeexPlayer*)inClientData;
    [audioController audioQueueOutputCallback:inAQ inBuffer:inBuffer];
}

void audioQueueIsRunningCallback(void *inClientData, AudioQueueRef inAQ,
                                 AudioQueuePropertyID inID) {

    SpeexPlayer *audioController = (__bridge SpeexPlayer*)inClientData;
    [audioController audioQueueIsRunningCallback];
}
@implementation SpeexPlayer

- (id)initWithStream:(RtmpMediaStream *)stream {
    if (self = [super initWithStream:stream]) {
        frame_size = 0;
        int enh = 1;
        started = NO;
        dec_state = speex_decoder_init(&speex_wb_mode);
        speex_decoder_ctl(dec_state, SPEEX_GET_FRAME_SIZE, &frame_size);
        speex_decoder_ctl(dec_state, SPEEX_SET_ENH, &enh);
        speex_decoder_ctl(dec_state, SPEEX_SET_SAMPLING_RATE, &sample_rate);
        speex_bits_init(&bits);
        AVAudioSession *audioSession = [AVAudioSession sharedInstance];
        [audioSession setCategory:AVAudioSessionCategoryPlayback error:nil];
        self.stream.audioDelegate = self;
    }
    return  self;
}
- (void)rtmpMediaStream:(RtmpMediaStream *)stream receivedAudioPacket:(RtmpPacket *)p {
    if (state != AUDIO_STATE_READY && state != AUDIO_STATE_PLAYING ) {
        [self createAudioQueue];
         [self startAudio];
    }
    static AVPacket packet;

    if (state != AUDIO_STATE_PLAYING && isStarted) {

    }
    isStarted = YES;
    if ([self encodePacket:p intoAVPacket:&packet]) {
        [self encodeRTMP:p intoQueueBuffer:audioQueueBuffer];
        av_free_packet(&packet);
        AudioTimeStamp bufferStartTime;
        AudioQueueGetCurrentTime(audioQueue, NULL, &bufferStartTime, NULL);
        bufferStartTime.mSampleTime += ((Float64)packet.dts)/1000;
        bufferStartTime.mFlags = kAudioTimeStampSampleTimeValid;
        tempTimeStamp.mSampleTime = bufferStartTime.mSampleTime - 5.0f;
        tempTimeStamp.mFlags = kAudioTimeStampSampleTimeValid;
        //NSLog(@"timestamp = %@",bufferStartTime);
        //[self enqueueBuffer:audioQueueBuffer withTimestamp:bufferStartTime];
       [self enqueueBuffer:audioQueueBuffer];
    }
}


- (BOOL)createAudioQueue {
    state = AUDIO_STATE_READY;
    AudioStreamBasicDescription audioStreamBasicDesc;
    audioStreamBasicDesc.mFormatID = kAudioFormatLinearPCM;
    audioStreamBasicDesc.mSampleRate = 8000;
    audioStreamBasicDesc.mFormatFlags = kLinearPCMFormatFlagIsSignedInteger |
    kLinearPCMFormatFlagIsPacked | kAudioFormatFlagsCanonical;
    audioStreamBasicDesc.mBytesPerPacket = 2;
    audioStreamBasicDesc.mFramesPerPacket = 1;
    audioStreamBasicDesc.mBytesPerFrame = sizeof(AudioSampleType);
    audioStreamBasicDesc.mChannelsPerFrame = 1 ;
    audioStreamBasicDesc.mBitsPerChannel = 8 * sizeof(AudioSampleType);
    audioStreamBasicDesc.mReserved = 0;
        OSStatus status = AudioQueueNewOutput(&audioStreamBasicDesc, audioQueueOutputCallback, (__bridge void*)self,
                                          NULL, NULL, 0, &audioQueue);
    if (status != noErr) {
        NSLog(@"Could not create new output.");
        return NO;
    }

    status = AudioQueueAddPropertyListener(audioQueue, kAudioQueueProperty_IsRunning,
                                           audioQueueIsRunningCallback, (__bridge void*)self);
    if (status != noErr) {
        NSLog(@"Could not add propery listener. (kAudioQueueProperty_IsRunning)");
        return NO;
    }
    status = AudioQueueAllocateBufferWithPacketDescriptions(audioQueue,
                                                                audioStreamBasicDesc.mSampleRate * kAudioBufferSeconds / 8,
                                                                audioStreamBasicDesc.mSampleRate * kAudioBufferSeconds / audioStreamBasicDesc.mFramesPerPacket + 1,
                                                                &audioQueueBuffer);
    //AudioQueueAllocateBufferWithPacketDescriptions(audioQueue, 400, 500, &audioQueueBuffer);
    //status = AudioQueueAllocateBuffer(audioQueue, audioStreamBasicDesc.mSampleRate * kAudioBufferSeconds / 8, &audioQueueBuffer);

    if (status != noErr) {
        NSLog(@"Could not allocate buffer.");
        return NO;
    }
    return YES;
}
- (void)startAudio {
    if (started) {
        AudioQueueStart(audioQueue, NULL);
    } else {
        [self startQueue];
    }
    state = AUDIO_STATE_PLAYING;
}

- (void)stopAudio{
    if (started) {
        AudioQueueStop(audioQueue, YES);
        state = AUDIO_STATE_STOP;
        if (!self.stream.videoDelegate) {
            [self.stream pause];
        }
    }
}

- (void)play {
    if (!self.stream.videoDelegate) {
        [self.stream play];
    }
}

- (void)pause {
    if (started) {
        state = AUDIO_STATE_PAUSE;
        AudioQueuePause(audioQueue);
        AudioQueueReset(audioQueue);
        if (!self.stream.videoDelegate) {
            [self.stream pause];
        }
    }
}
- (OSStatus)startQueue {
    OSStatus status = noErr;
    if (!started) {
        status = AudioQueueStart(audioQueue, NULL);
        if (status == noErr) {
            started = YES;
        }
        else {
            NSLog(@"Could not start audio queue.");
        }
    }
    return status;
}

- (void)removeAudioQueue {
    [self stopAudio];
    started = NO;
    AudioQueueFreeBuffer(audioQueue, audioQueueBuffer);
    AudioQueueDispose(audioQueue, YES);
}


-(AudioQueueBufferRef)encodeRTMP:(RtmpPacket*)packet intoQueueBuffer:(AudioQueueBufferRef)buffer {
    buffer->mAudioDataByteSize = 0;
    buffer->mPacketDescriptionCount = 0;
    NSData *data = packet.m_payload;
    NSData * subdata = [data subdataWithRange:NSMakeRange(1, data.length-1)];
    char *c = (char *)subdata.bytes;
    speex_bits_read_from(&bits, c,(int)subdata.length);
    short *outTemp = (short *)calloc(1, frame_size * 2);
    speex_decode_int(dec_state, &bits, outTemp);
    NSData *audioData = [NSData dataWithBytes:outTemp length:frame_size];
    if (buffer->mAudioDataBytesCapacity - buffer->mAudioDataByteSize >= audioData.length) {
        NSLog(@"true");
        memcpy((uint8_t *)buffer->mAudioData + buffer->mAudioDataByteSize, outTemp, audioData.length);
        buffer->mPacketDescriptions[buffer->mPacketDescriptionCount].mStartOffset = buffer->mAudioDataByteSize;
        buffer->mPacketDescriptions[buffer->mPacketDescriptionCount].mDataByteSize = (int)audioData.length;
        buffer->mPacketDescriptions[buffer->mPacketDescriptionCount].mVariableFramesInPacket = (int)audioData.length;
        buffer->mAudioDataByteSize += audioData.length;
        buffer->mPacketDescriptionCount++;
    }
    return buffer;
}

- (OSStatus)enqueueBuffer:(AudioQueueBufferRef)buffer{
           // withTimestamp:(AudioTimeStamp)timestamp {
    OSStatus status = noErr;

    NSLock * decodeLock = [[NSLock alloc] init];
    [decodeLock lock];

    if (buffer->mPacketDescriptionCount > 0) {
       //status = AudioQueueEnqueueBufferWithParameters(audioQueue, buffer, 0, NULL, 0, 0, 0, NULL, &timestamp, &timestamp);
       status = AudioQueueEnqueueBuffer(audioQueue, buffer, 0, NULL);
        if (status != noErr) {
            NSLog(@"Could not enqueue buffer.");
        }
    }
    else {
        AudioQueueStop(audioQueue, NO);
    }

    [decodeLock unlock];
    return status;
}
- (void)audioQueueOutputCallback:(AudioQueueRef)inAQ inBuffer:(AudioQueueBufferRef)inBuffer {
    if (state == AUDIO_STATE_PLAYING) {
        //[self enqueueBuffer:audioQueueBuffer withTimestamp:tempTimeStamp];
       [self enqueueBuffer:inBuffer ];
    }
}

- (void)audioQueueIsRunningCallback {
    UInt32 isRunning;
    UInt32 size = sizeof(isRunning);
    OSStatus status = AudioQueueGetProperty(audioQueue, kAudioQueueProperty_IsRunning, &isRunning, &size);
    if (status == noErr && !isRunning && state == AUDIO_STATE_PLAYING) {
        state = AUDIO_STATE_STOP;
    }
}

@end

I am enqueueing the audio queue buffer when the audio stream packets are received in the method - (void)rtmpMediaStream:(RtmpMediaStream *)stream receivedAudioPacket:(RtmpPacket *)p and also in the callback method - (void)audioQueueOutputCallback:(AudioQueueRef)inAQ inBuffer:(AudioQueueBufferRef)inBuffer . When AudioQueue plays the incoming stream, I am not able to listen to the proper audio. All I can hear is robotic voice and the tone of the audio is also very manly in all test cases. I cannot figure where I am going wrong. I cannot understand whether the problem is with the decoding part or playing audio using Audio using AudioQueues. Can someone suggest a solution?

`audioStreamBasicDesc.mFormatFlags = !kLinearPCMFormatFlagIsBigEndian | kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;` what does the exclamation mark mean ? — KudoCC, Apr 17 '14 at 08:24
If you want LittleEndian, just omit `kLinearPCMFormatFlagIsBigEndian`. `audioStreamBasicDesc.mFormatFlags = kLinearPCMFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsPacked;` — KudoCC, Apr 17 '14 at 09:05
ok..Can you identify something else that could create any issue? — user2955351, Apr 17 '14 at 09:15
Just fill `AudioStreamBasicDescription` with correct value. Are you sure `mSampleRate` is 11025 and `mChannelsPerFrame` is 1 ? — KudoCC, Apr 17 '14 at 09:22
Yes the Speex stream has the sample rate of 11kHz and is mono — user2955351, Apr 17 '14 at 09:27
Check `AudioStreamBasicDescription` in the [link](https://developer.apple.com/library/mac/documentation/musicaudio/reference/CoreAudioDataTypesRef/Reference/reference.html). mBytesPerFrame should be `sizeof (AudioSampleType)`. — KudoCC, Apr 17 '14 at 09:34
It is defined in CoreAudioTypes.h , and it's `typedef SInt16 AudioSampleType;` in my environment. — KudoCC, Apr 17 '14 at 09:45
I am sorry.. I didn't get much of what's given in the documentation.How can I set AudioSampleType in my app? Just create a `typedef`? — user2955351, Apr 17 '14 at 09:46
That link talks some detail information on each field of AudioStreamBasicDescription. — KudoCC, Apr 17 '14 at 09:55
and replace the value like this? `audioStreamBasicDesc.mBytesPerFrame = AudioSampleType;` — user2955351, Apr 17 '14 at 09:56
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/50863/discussion-between-kudocc-and-user2955351) — KudoCC, Apr 17 '14 at 09:57

Decoding Speex audio and playing decoded PCM data using AudioQueues

0 Answers0