Media foundation: Writing G711 PCMU to mp4

Question

We have a requirement (Windows UWP app) to store audio and video data received from cameras(RTP packets) in mp4.Video format is h264 and audio is g711 pcmu

We are using media foundation(c# using MF .Net) sinkWriter to write video data to mp4 which works fine. I would like to know how to write audio samples to mp4. I tried the following way:

private void SetupAudioMediaType(out IMFMediaType mediaType, in Guid audioSubType)
{    
    HResult hr = HResult.S_OK;

    hr = MFExtern.MFCreateMediaType(out mediaType);
    if (!hr.Succeeded())
    {
        Debug.Fail("MFCreateMediaType for audio failed " + hr.ToString());
        return;
    }

    hr = mediaType.SetGUID(MFAttributesClsid.MF_MT_MAJOR_TYPE, MFMediaType.Audio);
    if (!hr.Succeeded())
    {
        Debug.Fail("Set MF_MT_MAJOR_TYPE media-out failed " + hr.ToString());
        return ;
    }
    hr = mediaType.SetGUID(MFAttributesClsid.MF_MT_SUBTYPE, audioSubType);
    if (!hr.Succeeded())
    {
        Debug.Fail("Set MF_MT_SUBTYPE media-out failed " + hr.ToString());
        return ;
    }

    hr = mediaType.SetUINT32(MFAttributesClsid.MF_MT_AUDIO_SAMPLES_PER_SECOND, 48000);
    if (!hr.Succeeded())
    {
        Debug.Fail("Set MF_MT_AUDIO_SAMPLES_PER_SECOND media-out failed " + hr.ToString());
        return 
    }

    hr = mediaType.SetUINT32(MFAttributesClsid.MF_MT_AUDIO_NUM_CHANNELS, 1);
    if (!hr.Succeeded())
    {
        Debug.Fail("Set MF_MT_AUDIO_NUM_CHANNELS media-out failed " + hr.ToString());
        return ;
    }

    hr = mediaType.SetUINT32(MFAttributesClsid.MF_MT_AUDIO_BITS_PER_SAMPLE, 8);
    if (!hr.Succeeded())
    {
        Debug.Fail("Set MF_MT_AUDIO_BITS_PER_SAMPLE media-out failed " + hr.ToString());
        return ;
    }

    return errorCode;
}



private MultiplexerErrorCode SetupAudio()
{
    HResult hr = HResult.S_OK;

    IMFMediaType mediaTypeOut = null;
    IMFMediaType mediaTypeIn = null;

    SetupAudioMediaType(out mediaTypeOut, MFMediaType.AAC);  // or mp3 (MP4 in windows support mp3 or aac)
    if (errorCode != MultiplexerErrorCode.Success)
    {
        Debug.Fail("setupAudioMediaType output failed:", errorCode.ToString());
    }
    else
    {
        hr = sinkWriter.AddStream(mediaTypeOut, out audioStreamIndex);
        if (!hr.Succeeded())
        {
            Debug.Fail("AddStream  audio  failed " + hr.ToString());
        }
        else
        {
            Guid PcmuAudioSubType = (new FourCC(7,0,0,0)).ToMediaSubtype();   //PCMU
            SetupAudioMediaType(out mediaTypeIn, PcmuAudioSubType);
            hr = sinkWriter.SetInputMediaType(audioStreamIndex, mediaTypeIn, null);
            if (!hr.Succeeded())
            {
                Debug.Fail("SetInputMediaType audio  failed " + hr.ToString());
            }
        }
    }

    return ;
}

SetInputMediaType returns error MF_E_INVALIDMEDIATYPE. From my analysis following are the reasons for the error

1) I think PCMU input type is not supported. It should be PCM. Is this understanding corect? If so does this mean I have to decode PCMU to PCM.If so is there any windows c# API which does this ? And once this decode is done , what are the bits per sample in the ouput pcm. Is it 16?

2)Even if I provide pcm as input type, SetInputMediaType returns MF_E_INVALIDMEDIATYPE error. Is it because the aac encoder supports only sampling rate of 44.1 & 48Khz. (mp3 supports 32 Khz,...). If my understanding is correct, how do I overcome the issue. Should I up-sample. If so how?

3)Is there any simpler way to write pcmu(8000 samples per second, 8 bits per sample) to mp4 along with the video frames

If you're using C# then the quick way to decode the audio is to use NAudio. To gain an in depth understanding see https://www.codeproject.com/Articles/501521/How-to-convert-between-most-audio-formats-in-NET. — sipsorcery, Feb 15 '20 at 21:36

mofo77 · Answer 1 · 2020-02-13T20:12:25.403

2

For 1) and 3) read this : muxing-only-audio-into-mp4-ffmpeg-failed

For 2)

decode G711 to PCM (CLSID_MULawCodecWrapper : 92B66080-5E2D-449E-90C4-C41F268E5514)
use an audio resampler (Audio Resampler DSP)
encode PCM resampled to AAC (AAC Encoder)

Perhaps CLSID_MULawCodecWrapper is able do to resampling. Perhaps you will have audio/video synchronization problems.

edited Feb 13 '20 at 20:12

answered Feb 13 '20 at 20:06

mofo77

1,447
9
17

Thanks a lot. I will check CLSID_MULawCodecWrapper and also check audio resampler. I wanted to check if my code works if I pass PCM input (48000Hz, 16 bits per sample). My code works and I get playable audio(.m4a file aac codec). But if I change to mp3(.mp3 file and audio subtype to .mp3), I get MF_E_INVALIDMEDIATYPE . Why does mp3 media type give error. From document I see it supports PCM, sampling rate to 48000Hz, bits per sample 16, channel count 1 . Should I provide any more attributes. What is wrong in my configuration – karthik vr Feb 14 '20 at 11:46
Can Windows Media Foundation be used to fix corrupted video by re-encoding them? – Michael Haephrati Jun 26 '22 at 09:30

Media foundation: Writing G711 PCMU to mp4

1 Answers1