0

I am trying to build a opensource video system in android, since we have no access to the data in a closed system. In this system, we can modify the raw data captured by camera.

I used MediaCodec and MediaMux to do the video data encoding and muxing job, and that works. But I have no idea about the audio part. I used onFramePreview to get each frame and do the encoding/muxing work by frame. But how do I do the audio recording at the same time(I mean capturing the audio by frame, encode it and send the data to the MediaMux).

I've done some research. It seems that we use audiorecorder to get the raw data of audio. But audiorecorder does a constant recording job, I don't think it can work.

Can anyone give me a hint? Thank you!

Brendon Tsai
  • 1,267
  • 1
  • 17
  • 31

1 Answers1

1

Create audioRecorder like this: 

private AudioRecord getRecorderInstance() {
    AudioRecord ar = null;
    try {
        //Get a audiorecord
        int N = AudioRecord.getMinBufferSize(8000,AudioFormat.CHANNEL_IN_MONO,AudioFormat.ENCODING_PCM_16BIT);            
        ar = new AudioRecord(AudioSource.MIC, 8000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, N*10);
    } 
    catch (Exception e) {

    }
    return ar; //Returns null if mic is unavailable
}

Prepare and send the data for the encoding and muxing later like this is separate thread:

public class MicrophoneInput implements Runnable {
    @Override
    public void run() {
        // Buffer for 200 milliseconds of data, e.g. 400 samples at 8kHz.
        byte[] buffer200ms = new byte[8000 / 10];

        try {
                while (recording) {
                    audioRecorder.read(buffer200ms, 0, buffer200ms.length);

                    //process buffer i.e send to encoder
                    //don't forget to set correct timestamps synchronized with video
                }
        } 
        catch(Throwable x) {        
                //  
        }
    }
}
Marlon
  • 1,473
  • 11
  • 13
  • Thank you, @Marlon. I've tried it but I meet some troubles. Where should I put the function audioRecord.startRecording? I use the callback function onPreviewFrame to get the data of every frame and encode/mux these datas. I've tried to put startRecording in onPreviewFrame and before onPreviewFrame, but either works. Besides, I used AsyncTask to do the encode background(onPreivewFrame call that task by frame). If I open another thread for audioRecord, how can I provide the audio by frame for the muxer? Thank you! – Brendon Tsai Feb 25 '14 at 03:01
  • 1
    I have audioRecord.startRecording right after camera.startPreview(). The main idea is that video is encoded and sent to muxer in camera's thread. Audio chain lives in own thread\Runnable which is looped in while{} and reads audio samples from audioRecorder when samples ready (200ms here) and sent to encoder and muxer after. Runnable is created\started together with audioRecord start: MicrophoneInput mi = new MicrophoneInput(); audioThread = new Thread(mi); – Marlon Feb 25 '14 at 13:17
  • 1
    One more thought: a sync object could be added to the onPreviewFrame, to push audio thread on every video frame came in. So every video frame would trigger audio buffer send to encoder in separate thread. But it will need more precise buffer allocation for audioRecorder – Marlon Feb 25 '14 at 13:22
  • Thank you. And thank you for solve both my problems. I am confused about how to send the buffer from this thread to the onPreiewFrame. And How do we make it async? Do we just make it send every 200ms(control by time) or something else? Could you provide more details? Thank you! – Brendon Tsai Feb 26 '14 at 09:22
  • 1
    you need to create 2 instances of mediacodec, one for audio, another for video, configure them correctly. after it send the data to videoencoder from onPrevieFrame, to audioencoder from run() in MicrophoneInput. Simply use the loop to get data from audioRecorder and send to encoder. A room for performance improvements is there for sure. And always send correct timestamps to encoder input - muxer makes syncronization basing on it – Marlon Feb 26 '14 at 11:11
  • 1
    from my understading audioRecorder works in the next way: it fills internal buffers in internal thread, when you ask for audioRecorder.read(buffer200ms, 0, buffer200ms.length); it just tries to copy 200ms from internal buffer to buffer200ms. if less then 200ms are in internal buffer it will return the actual copied data size. may be it is better to allocate 1 second buffer and make read in some less frequent way – Marlon Feb 26 '14 at 11:16
  • Thank you for elaborate it. So we write the audio record and encode function in run(), and video record and encode function in the onPreviewFrame. We make sure the timestamps for the encoder is right, and then we send the data to Muxer. Thank you! – Brendon Tsai Feb 26 '14 at 11:37