Video Transcode with Android MediaCodec

Question

Struggling with Android MediaCodec, I'm looking for a straight forward process to change the resolution of a video file in Android.

For now I'm trying a single thread transcoding method that makes all the work step by step so I can understand it well, and at high level it looks as follows:

public void TranscodeVideo()
{
    // Extract
    MediaTrack[] tracks = ExtractTracks(InputPath);

    // Decode
    MediaTrack videoTrack = tracks.Where(o => o.IsVideo).FirstOrDefault();
    MediaTrack rawVideoTrack = DecodeTrack(videoTrack);

    // Edit?
    // ResizeVideoTrack(rawVideoTrack);

    // Encode
    MediaFormat newFormat = MediaHelper.CreateVideoOutputFormat(videoTrack.Format);
    MediaTrack encodeVideodTrack = EncodeTrack(rawVideoTrack , newFormat);

    // Muxe
    encodeVideodTrack.Index = videoTrack.Index;
    tracks[Array.IndexOf(tracks, videoTrack)] = encodeVideodTrack;
    MuxeTracks(OutputPath, tracks);
}

Extraction works fine, returning a track with audio only and a track with video only. Muxing works fine combining again two previous tracks. Decoding works but I don't know how to check it, the raw frames on the track weight much more than the originals so I assume that it's right.

Problem

The encoder input buffer size is smaller than the raw frames size, and also related to the encoding configured format, so I assume that I need to resize the frames in some way but I don't find anything useful. I'm correct on this? I'm missing something? What is the way to go resizing Raw video frames? Any help? :S

PD

• Maybe you will notice that I'm using C# (Xamarin.Android) for more fun. But the underlaying API is of course Java.
• I'm using ByteBuffers, not Surfaces because it seems easier. I will be the next step using surfaces, any advice is welcome.
• I know that the single thread process is highly inefficient, but makes it simple. It will be another next step to connect the decoder output buffer to the encoder input buffer.
• I digged through PhilLab, Grafika and Bigflake examples but nothing seems to be very useful for me.
• Avoiding to use ffmpeg on Android.

Thank you everyone for your time.

Answer 1

Going off of the comment above to implement libVLC

Add this to your app root's build.gradle

allprojects {
    repositories {
       ...
       maven {
           url 'https://jitpack.io'
       }
   }
}

Add this to your dependent app's build.gradle

dependancies {
    ...
    implementation 'com.github.masterwok:libvlc-android-sdk:3.0.13'
}

Here is an example of loading an RTSP stream as an activity

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.camera_stream_layout);


    // Get URL
    this.rtspUrl = getIntent().getExtras().getString(RTSP_URL);
    Log.d(TAG, "Playing back " + rtspUrl);

    this.mSurface = findViewById(R.id.camera_surface);
    this.holder = this.mSurface.getHolder();

    ArrayList<String> options = new ArrayList<>();
    options.add("-vvv"); // verbosity
    //Add vlc transcoder options here

    this.libvlc = new LibVLC(getApplicationContext(), options);
    this.holder.setKeepScreenOn(true);
    //this.holder.setFixedSize();

    // Create media player
    this.mMediaPlayer = new MediaPlayer(this.libvlc);
    this.mMediaPlayer.setEventListener(this.mPlayerListener);

    // Set up video output
    final IVLCVout vout = this.mMediaPlayer.getVLCVout();
    vout.setVideoView(this.mSurface);

    //Set size of video to fit app screen
    DisplayMetrics displayMetrics = new DisplayMetrics();
    getWindowManager().getDefaultDisplay().getMetrics(displayMetrics);

    ViewGroup.LayoutParams videoParams = this.mSurface.getLayoutParams();
    videoParams.width = displayMetrics.widthPixels;
    videoParams.height = displayMetrics.heightPixels;

    vout.setWindowSize(videoParams.width, videoParams.height);
    vout.addCallback(this);
    vout.attachViews();

    final Media m = new Media(this.libvlc, Uri.parse(this.rtspUrl));
    //Use this to add transcoder options m.addOption("vlc transcode options here");
    this.mMediaPlayer.setMedia(m);
    this.mMediaPlayer.play();
}

Here is the documentation of vlc transcoder options

https://wiki.videolan.org/Documentation:Streaming_HowTo_New/

Answer 2

You are right, the input buffer size of the encoder is smaller because it expects input to be of the specified dimensions. The encoder only, like the name suggests, encodes.

I read your question as more of a "why" than a "how" question so i'll only point you to where you'll find the "why's"

The decoded frame is a YUV image (is suggest to quickly skim through the wikipedia article), usually NV21 if i'm not mistaken but might be different from device to device. To do this i suggest you use a library as every every plane of the image needs to be scaled down differently and it usually takes care of filtering.Check out libYUV. If you are interested in the actual resizing algorithms check out this and for implementations this.

If you are not required to handle the decoding and encoding with bytebuffers, i suggest to use a surface as you already mentioned. It has multiple benefits over decoding to bytebuffers.

• More memory efficient as there is no copy between the native buffer and app allocated buffer, the native buffers are simply geting swapped from and to the surface.
• If you plan to render the frame, be it for resizing or displaying, it can be done by the devices graphic processor. On how to do that check out BigFlakes DecodeEditEncode test.

In hope this answers some of your questions.