Converting RGB to YUV, + ffmpeg

Question

I am trying the following to record a live video from my Flash/AIR application:

1. I take a "screenshot" (BitmapData from stage) each frame.

2. I convert each pixel to yuv format like this (V2):

       var file :File = new File(_appUrl + "/creation/output.raw");
       var fs :FileStream = new FileStream();
       fs.open(file, FileMode.WRITE);
       var finalY :ByteArray = new ByteArray();
       var finalU :ByteArray = new ByteArray();
       var finalV :ByteArray = new ByteArray();
       var rect :Rectangle = new Rectangle(0, 0, 600, 700);
       var pixels :ByteArray;
       var pixel :uint;
       var r :uint;
       var g :uint;
       var b :uint;
       _screenBuffer = _screenBuffer.reverse();
       while (_screenBuffer.length > 0)
       {
           pixels = BitmapData(_screenBuffer.pop()).getPixels(rect);
           pixels.position = 0;
           // Convert and save each pixel
           for (var x:int = 0; x < 600; x++)
           {
               for (var y:int = 0; y < 700; y++)
               {
                   // Convert to yuv
                   pixel = pixels.readUnsignedInt();
                   r = pixel >> 16 & 0xff;
                   g = pixel >> 8 & 0xff;
                   b = pixel & 0xff;
                       // Y' is written for each pixel
                   finalY.writeByte(0.257 * r + 0.504 * g + 0.098 * b + 128);
                       // U and V are written once per 2x2 pixel block
                   if (x % 2 == 0 && y % 2 == 0)
                   {
                       finalU.writeByte(-0.148 * r - 0.291 * g + 0.439 * b + 128);
                       finalV.writeByte(0.439 * r - 0.368 * g - 0.071 * b + 128);
                   }
               }
           }
       }
       // Write the converted bytes to the file
       finalY.position = 0;
       finalU.position = 0;
       finalV.position = 0;
       fs.writeBytes(finalY, 0, finalY.length);
       fs.writeBytes(finalU, 0, finalU.length);
       fs.writeBytes(finalV, 0, finalV.length);
       fs.close();
   

3. I use the following line of ffmpeg to convert the raw file to a video:

   ffmpeg -r 30 -pix_fmt yuv420p -s 600x700 -vcodec rawvideo -f rawvideo -i output.raw -y test.mp4 
   

A video is created, but it is simply a mess, barely resembling what was recorded. I know that the capturing process works, as I have tried the same BitmapData "screenshots" with the SimpleFlvWriter.

So, either something is wrong with my conversion or with the ffmpeg line, but I have no idea.

This is what ffmpeg outputs when creating the video, maybe it can help someone:

libavutil      51. 39.100 / 51. 39.100
libavcodec     54.  3.101 / 54.  3.101
libavformat    54.  1.100 / 54.  1.100
libavdevice    53.  4.100 / 53.  4.100
libavfilter     2. 62.101 /  2. 62.101
libswscale      2.  1.100 /  2.  1.100
libswresample   0.  7.100 /  0.  7.100
libpostproc    52.  0.100 / 52.  0.100
[rawvideo @ 01D39FC0] Estimating duration from bitrate, this may be inaccurate
Input #0, rawvideo, from 'output.raw':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 600x700, 30 tbr,
30 tbn, 30 tbc
[buffer @ 01D3FEC0] w:600 h:700 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 @ 0375DB80] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE 4.2 AVX
[libx264 @ 0375DB80] profile High, level 3.1
[libx264 @ 0375DB80] 264 - core 120 r2146 bcd41db - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'test.mp4':
  Metadata:
    encoder         : Lavf54.1.100
    Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 600x700, q=-1--1, 30 tbn, 30 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo -> libx264)
Press [q] to stop, [?] for help
Truncating packet of size 630000 to 1
frame=   48 fps=  0 q=-1.0 Lsize=     157kB time=00:00:01.53 bitrate= 837.3kbits/s
video:156kB audio:0kB global headers:0kB muxing overhead 0.687626%
[libx264 @ 0375DB80] frame I:3     Avg QP:23.15  size: 23480
[libx264 @ 0375DB80] frame P:38    Avg QP:28.80  size:  2169
[libx264 @ 0375DB80] frame B:7     Avg QP:29.61  size:   833
[libx264 @ 0375DB80] consecutive B-frames: 79.2%  4.2%  0.0% 16.7%
[libx264 @ 0375DB80] mb I  I16..4: 41.4%  6.2% 52.4%
[libx264 @ 0375DB80] mb P  I16..4: 10.6%  3.3%  0.9%  P16..4: 68.4%  1.3%  1.2% 0.0%  0.0%    skip:14.2%
[libx264 @ 0375DB80] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8: 13.3%  2.2%  0.7% direct: 1.9%  skip:81.9%  L0:51.6% L1:47.4% BI: 1.0%
[libx264 @ 0375DB80] 8x8 transform intra:16.7% inter:31.2%
[libx264 @ 0375DB80] coded y,uvDC,uvAC intra: 14.7% 25.5% 22.3% inter: 1.0% 4.1% 3.4%
[libx264 @ 0375DB80] i16 v,h,dc,p: 87% 11%  2%  0%
[libx264 @ 0375DB80] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  3% 18% 75%  1%  0%  1%  1% 0%  0%
[libx264 @ 0375DB80] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu:  6% 74% 12%  1%  1%  1%  2% 1%  2%
[libx264 @ 0375DB80] i8c dc,h,v,p: 51% 45%  4%  1%
[libx264 @ 0375DB80] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0375DB80] ref P L0:  4.6%  0.4% 94.6%  0.3%
[libx264 @ 0375DB80] ref B L0: 96.0%  4.0%
[libx264 @ 0375DB80] ref B L1: 96.5%  3.5%
[libx264 @ 0375DB80] kb/s:793.39

I'm not really a codec expert (just starting ;)), so I don't know what to make of most of that.

Here is a zip that contains one of the frames and the video output. What should be visible is a green smiling pear, without any artifacts. Remember the size is 600x700 and the format yuv420. Best to view such raw image files with irfanview, IMO. Don't mind the noise, its from pushing against my microphone ;)

Answer 1

You use the pixel format

-pix_fmt yuv420p 

which in memory is

Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 U1234 U5678 V1234 V5678

ie there is U and one V for each square of 4 pixels. Furthermore, you have all the Y first, then all the U and then all the V.

It seems you are writing

Y1 U1 V1 Y2 U2 V2 ...

There are multiple pixel formats. If you want to use yuv420p, you need to stick to the memory layout. Check the Wikipedia entry on yuv420p