0

I'm trying to decode H264 frames, sent by my backend, like this:

/*
packet = {
  type: "frame",
  keyframe: <Boolean>,
  pts: <BigInt>,
  data: <ArrayBuffer/Uint8Array>
}
*/
const chunk = new EncodedVideoChunk({
  type: packet.keyframe === false ? 'delta' : 'key',
  timestamp: 0,
  data: data
});
console.debug("CHUNK");
this.decoder.decode(chunk);

The decoder looks like this:

this.decoder = new VideoDecoder({
  output: (frame) => {
    console.debug("DECODE")
    <...>
  error: (error) => {
    console.error(error);      
  }
});

The problem I have is that my log DECODE is never printed, whereas CHUNK is, but at the same time, there are also no errors thrown.

If anyone has an idea on what else I could try, I'd be very grateful.

Tabea
  • 95
  • 9

3 Answers3

1

"I'm trying to decode H264 frames, sent by my backend, like this..."

The decoder setup depends on your input H.264 data:
Assuming it provides data in AnnexB format whose codec name is known (eg: "avc1.42C01E").

You can find the part after "avc1." by extracting the first 3 byte values in your SPS and converting them each into a hex string.

AnnexB means...

  • Each frame starts with byte sequence: 00 00 00 01.
  • If it's a first frame, then it must also include the SPS and PPS data.

Example H.264 keyframe bytes (from AnnexB file):

00 00 00 01 67 42 C0 1E DD EC 04 40 00 00 03 00 40 00 00 0F 03 C5 8B E0 00 00 00 01 68 CE 0F 2C 80 00 00 01 65 88 84 04 BC 46 28 00 0C 7D 47 00 01 7D 78 E0 00 22 3D 80

Below is the minimum required code to decode the above H.264 keyframe bytes.

  • Future frames will only need to update new frame data using:

    myOutputFrameData.data = new Uint8Array( some frame bytes )

  • Also update the frame "type" which can be "key" (IDR & I frames) or "frame" (P & B frames):

    myOutputFrameData.type = "key"

  • Then decode new frame data with:

    const chunk_frame = new EncodedVideoChunk( myOutputFrameData );
    decoder.decode( chunk_frame );

Example code for decoding some H.264 frame...

<!DOCTYPE html>
<html>
<body>
<!-- Canvas to display a decoded frame -->
<canvas id="myCanvas" width="320" height="240">
</canvas>

<br>A decoded frame (eg: yellow fill)

<script>
    
//# for displaying decoded frame in Canvas
const myCanvas = document.getElementById("myCanvas");
const ctx = myCanvas.getContext("2d");

//# STRING for codec format... 
str_codec_format = "annexb";

//# STRING for codec name... is: "avc1." + 0xAA + 0xBB + 0xCC
var str_codec_name = "avc1.42C01E";

///////////////////////////////////////////

//# AnnexB needs SPS + PPS at front then followed by keyframe..
let myFrameData_Key = new Uint8Array( 
                                        [   
                                            //# SPS (0x67 ... etc )
                                            0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0xC0, 0x1E, 0xDD, 0xEC, 0x04, 0x40, 0x00, 0x00, 0x03, 0x00, 0x40, 0x00, 0x00, 0x0F, 0x03, 0xC5, 0x8B, 0xE0, 
                                            
                                            //# PPS (0x68 ... etc )
                                            0x00, 0x00, 0x00, 0x01, 0x68, 0xCE, 0x0F, 0x2C, 0x80,
                                            
                                            //# Keyframe data (0x65 ... etc )                       
                                            0x00, 0x00, 0x00, 0x01, 0x65, 0x88, 0x84, 0x04, 0xBC, 0x46, 0x28, 0x00, 0x0C, 0x7D, 0x47, 0x00, 0x01, 0x7D, 0x78, 0xE0, 0x00, 0x22, 0x3D, 0x80 
                                        ] 
                                    );
                                    
//# setup decoder initialize...
let decoder_init = {
  output: ( videoFrame ) => {  handleFrame( videoFrame ); },
  error: ( evt ) => { console.log( evt.message ); },
};

//# setup decoder config...
let decoder_config = {};
decoder_config.codec = str_codec_name;
decoder_config.avc = { format: str_codec_format }
decoder_config.hardwareAcceleration = "prefer-hardware";

//# start Video decoder...
const decoder = new VideoDecoder( decoder_init );
decoder.configure( decoder_config );

//#  Object to send to Video decoder (holds frame data + frame settings)
let myOutputFrameData = { }; 
myOutputFrameData.type = "key";
myOutputFrameData.timestamp = 0;

//# Add frames bytes into Object sent to the VideoDecoder
myOutputFrameData.data =  new Uint8Array( myFrameData_Key );

/////# Try to decode a keyframe

//# convert Object into EncodedVideoChunk...
const chunk_frame = new EncodedVideoChunk( myOutputFrameData );

//# Decoder only accepts Objects from EncodedVideoChunk...
decoder.decode( chunk_frame );
  
async function handleFrame( input )
{
    alert("Decode success : Got Frame = " + input );

    //# draw frame inside a Canvas object
    ctx.drawImage(input, 0, 0, myCanvas.width, myCanvas.height);

    //# close frame and flush decoder (send to display)
    input.close(); //decoder.flush();
    await decoder.flush();
}

</script>

</body>
</html>
VC.One
  • 14,790
  • 4
  • 25
  • 57
1

A single decode() will usually not result in a frame being call-backed. The reason is (I assume) that the decoder doesn't know if the next decode() will be for a frame that is earlier in display order (the order of frames in the byte stream may be different from the order in which they are displayed, and the VideoDecoder outputs frames in display order (presentation order).

So the solution is to let the VideoDecoder know that you're done by calling:

await this.decoder.flush()

(or this.decoder.flush().then(() => console.log("FLUSHED")) if it's not an async function)

Note that if you want to call this.decoder.close(), this must be done after the flush is complete (i.e. either await the flush promise, or put it in the then() section.

Claude
  • 8,806
  • 4
  • 41
  • 56
1

Like mentioned below, the decoder will not output untill it was fed with sufficient frames. Equally, when you feed it with chunks corresponding to x number of frames and then you stop, you'll verify that it will output x-2 frames and then stop outputing. I believe it holds the last two frames in memory to use them to decode the next frames it will recieve. For this reason, the proper way to know if the chunk that was fed was correctly decoded is by monitoring videoDecoder.ondeque event. It will fire even in the cases when there is no output, as long as the decoding process didn't fail.

pedrobroese
  • 46
  • 3