How to encode and decode audio using opus

Question

I am trying integrate opus into my application, the encode and decode function returns positive value which means successfully, but the output audio can't play. Raw audio data can play as well. Here is how I encode data. I use 4 bytes prefix to separate from each packet.

self.encoder = opus_encoder_create(24000, 1, OPUS_APPLICATION_VOIP, &opusError);
opus_encoder_ctl(self.encoder, OPUS_SET_BANDWIDTH(OPUS_BANDWIDTH_SUPERWIDEBAND));



- (void) encodeBufferList:(AudioBufferList *)bufferList {
    BOOL success = TPCircularBufferProduceBytes(_circularBuffer, bufferList->mBuffers[0].mData, bufferList->mBuffers[0].mDataByteSize);
    if (!success) {
        NSLog(@"insufficient space in circular buffer!");
    }

    if (!_encoding) {
            _encoding = YES;

            dispatch_async(self.processingQueue, ^{
                [self startEncodingLoop];
            });
    }
}


-(void)startEncodingLoop
{
    int32_t availableBytes = 0;
    opus_int16 *data = (opus_int16*)TPCircularBufferTail(_circularBuffer, &availableBytes);
    int availableSamples = availableBytes / _inputASBD.mBytesPerFrame;

    /*!
     *  Use dynamic duration
     */
//    int validSamples[6] = {2.5, 5, 10, 20, 40, 60}; // in milisecond
//    int esample = validSamples[0] * self.sampleRate / 1000;
//    for (int i = 0; i < 6; i++) {
//        int32_t samp = validSamples[i] * self.sampleRate / 1000;
//        if (availableSamples < samp) {
//            break;
//        }
//        esample = samp;
//    }

    /*!
     *  Use 20ms
     */
    int esample = 20 * self.sampleRate / 1000;

    if (availableSamples < esample) {
        /*!
         *  Out of data. Finish encoding
         */
        self.encoding = NO;
        [self.eDelegate didFinishEncode];
        return;
    }

//    printf("raw input value for packet \n");
//    for (int i = 0; i < esample * self.numberOfChannels; i++) {
//        printf("%d :", data[i]);
//    }

    int returnValue = opus_encode(_encoder, data, esample, _encoderOutputBuffer, 1000);

    TPCircularBufferConsume(_circularBuffer, esample * sizeof(opus_int16) * self.numberOfChannels);

//    printf("output encode \n");
//    for (int i = 0; i < returnValue; i++) {
//        printf("%d :", _encoderOutputBuffer[i]);
//    }

    NSMutableData *outputData = [NSMutableData new];
    NSError *error = nil;
    if (returnValue <= 0) {
        error = [OKUtilities errorForOpusErrorCode:returnValue];
    }else {
        [outputData appendBytes:_encoderOutputBuffer length:returnValue * sizeof(unsigned char)];
        unsigned char int_field[4];
        int_to_char(returnValue , int_field);
        NSData *header = [NSData dataWithBytes:&int_field[0] length:4 * sizeof(unsigned char)];
        if (self.eDelegate) {
            [self.eDelegate didEncodeWithData:header];
        }
    }

    if (self.eDelegate) {
        [self.eDelegate didEncodeWithData:outputData];
    }

    [self startEncodingLoop];
}

And here is decode function:

self.decoder = opus_decoder_create(24000, 1, &opusError);
opus_decoder_ctl(self.decoder, OPUS_SET_SIGNAL(OPUS_SIGNAL_VOICE));
opus_decoder_ctl(self.decoder, OPUS_SET_GAIN(10));


-(void)startParseData:(unsigned char*)data remainingLen:(int)len
{
    if (len <= 0) {
        [self.dDelegate didFinishDecode];
        return;
    }
    int headLen = sizeof(unsigned char) * 4;
    unsigned char h[4];
    h[0] = data[0];
    h[1] = data[1];
    h[2] = data[2];
    h[3] = data[3];

    int packetLen = char_to_int(h);
    data += headLen;
    packetLen = packetLen * sizeof(unsigned char) * self.numberOfChannels;
    [self decodePacket:data length:packetLen remainingLen:len - headLen];
}

-(void)decodePacket:(unsigned char*)inputData length:(int)len remainingLen:(int)rl
{
    int bw = opus_packet_get_bandwidth(inputData); //TEST: return OPUS_BANDWIDTH_SUPERWIDEBAND here
    int32_t decodedSamples = 0;

//    int validSamples[6] = {2.5, 5, 10, 20, 40, 60}; // in milisecond
    /*!
     *  Use 60ms
     */
    int esample = 60 * self.sampleRate / 1000;
//    printf("input decode \n");
//    for (int i = 0; i < len; i++) {
//        printf("%d :", inputData[i]);
//    }

    _decoderBufferLength = esample * self.numberOfChannels * sizeof(opus_int16);
    int returnValue = opus_decode(_decoder, inputData, len, _outputBuffer, esample, 1);
    if (returnValue < 0) {
        NSError *error = [OKUtilities errorForOpusErrorCode:returnValue];
        NSLog(@"decode error %@", error);
        inputData += len;
        [self startParseData:inputData remainingLen:rl - len];
        return;
    }
    decodedSamples = returnValue;

    NSUInteger length = decodedSamples * self.numberOfChannels;

//    printf("raw decoded data \n");
//    for (int i = 0; i < length; i++) {
//        printf("%d :", _outputBuffer[i]);
//    }

    NSData *audioData = [NSData dataWithBytes:_outputBuffer length:length * sizeof(opus_int16)];
    if (self.dDelegate) {
        [self.dDelegate didDecodeData:audioData];
    }
    inputData += len;
    [self startParseData:inputData remainingLen:rl - len];
}

Please help me to point out what I am missing. An example would be great.

Is there an example project where I can see how this code functions? — hungrxyz, Jul 23 '15 at 14:25
@sahara108 iam also facing similar issue did you find solution? — Aslam, May 19 '20 at 07:39
@Aslam yes. I posted my answer at the end of this page. You can check it. — sahara108, May 21 '20 at 07:50
@sahara108 i have small doubt in this could you share your skype id or mail id..so that i can explain my issue?? — Aslam, May 26 '20 at 02:47

score 5 · Answer 1 · answered May 29 '15 at 18:19

I think the problem is on the decode side:

You pass 1 as the fec argument to opus_decode(). This asks the decoder to generate the full packet duration's worth of data from error correction data in the current packet. I don't see any lost packet tracking in your code, so 0 should be passed instead. With that change your input and output duration should match.
You configure the decoder for mono output, but later use self.numberOfChannels in length calculations. Those should match or you may get unexpected behaviour.
OPUS_SET_SIGNAL doesn't do anything in opus_decoder_ctl() but it will just return OPUS_UNIMPLEMENTED without affecting behaviour.
Opus packets can be up to 120 ms in duration, so your limit of 60 ms could fail to decode some streams. If you're only talking to your own app that won't cause a problem the way you've configured it, since libopus defaults to 20ms frames.

Thanks for your help. The `fec` should be `0`. I have changed it. I found what the problem is, I have set the audio output is float but use `opus_encode` and `opus_decode`. Change to `opus_encode_float` and `opus_decode_float` fix my problem. — sahara108, Jun 01 '15 at 02:41

score 3 · Accepted Answer · answered Jun 01 '15 at 02:46

I found what the problem is. I have set the audio format is float kAudioFormatFlagIsPacked|kAudioFormatFlagIsFloat;. I should use opus_encode_float and opus_decode_float instead of opus_encode opus_decode. As @Ralph says, we should use fec = 0 in opus_decode. Thanks to @Ralph.

score 2 · Answer 3 · answered May 26 '15 at 18:33

2

One thing I notice is that you're treating the return value of opus_encode() as a number of samples encoded, when it's the number of bytes in the compressed packet. that means you're writing 50% or 75% garbage data from the end of _encoderOutputBuffer into your encoded stream.

Also make sure _encoderOutputBuffer has room for the hard-coded 1000 byte packet-length limit you're passing in.

answered May 26 '15 at 18:33

Ralph Giles

467
2
9

The `_encoderOutputBuffer` has room for 1000 byte. I allocate it before encoding: `self.encoderOutputBuffer = malloc(1000 * sizeof(unsigned char));` Can you explain why I am treating the `returnValue` as number of samples encoded? In console it print out 27 ~ 30 bytes. – sahara108 May 27 '15 at 02:11
You're right, sorry. I misread the `* sizeof(unsigned char)` as `* sizeof(opus_int16)`. – Ralph Giles May 29 '15 at 17:38

How to encode and decode audio using opus

3 Answers3