I'm trying to rotate a h264 video 90 degrees anticlockwise. The syntax for the Display Orientation Supplementary EnhancementInformation (SEI) is given as:
Which I first tried encoding as follows:
val prefix = byteArrayOf(0, 0, 0, 1)
val nalHeader = byteArrayOf(6) // 0 forbidden_zero_bit = 0, nal_ref_idc = 0, nal_unit_type = 6
val display = byteArrayOf(47 /* Display orientation type*/, 3 /*payload size*/)
val displayOrientationCancelFlag = "0" // u(1); Rotation information follows
val horFlip = "0" // hor_flip; u(1); Do not flip horizontally
val verFlip = "0" // ver_flip; u(1); Do not flip vertically
val anticlockwiseRotation = "0100000000000000" // u(16); value / 2^16 -> 90 degrees
val displayOrientationRepetitionPeriod = "010" // ue(v); Persistent till next video sequence
val displayOrientationExtensionFlag = "0" // u(1); No other value is permitted by the spec atm
val byteAlignment = "1"
The above is my kotlin code for generating the SEI. This is a static variable so I went for human readable version.
Dropping the Annex B start codes, the hex SEI is
06 2f 03 08 00 09
ffmpeg complained about this however, saying that my payload size was specified as 3 bytes (24 bits) but it only read 23 bits before reading the NAL stop bit. To fix this I padded my SEI payload by one zero bit to allow ffmpeg parse it successfully, and added the stop bit again with more byte alignment bits:
val byteAlignment = "010000000"
This in hex is
06 2f 03 08 00 08 80
When I add this before the first IDR NAL unit in my h264 rbsp, ffmpeg will accept it to convert to whatever format I ask, jpeg or mp4. The output is not rotated however. Playing it with ffplay also does not rotate it. I'm note quite sure what I'm doing wrong. ffprobe yields the following output:
[NULL @ 0x7fca03808a00] Opening 'sei2.264' for reading
[file @ 0x7fca01c289c0] Setting default whitelist 'file,crypto'
Probing h264 score:51 size:1502
Probing mp3 score:1 size:1502
[h264 @ 0x7fca03808a00] Format h264 probed with size=2048 and score=51
[h264 @ 0x7fca03808a00] Before avformat_find_stream_info() pos: 0 bytes read:1502 seeks:0 nb_streams:1
[AVBSFContext @ 0x7fca02808680] nal_unit_type: 7(SPS), nal_ref_idc: 3
[AVBSFContext @ 0x7fca02808680] nal_unit_type: 8(PPS), nal_ref_idc: 3
[AVBSFContext @ 0x7fca02808680] nal_unit_type: 6(SEI), nal_ref_idc: 0
[AVBSFContext @ 0x7fca02808680] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7fca0381d400] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7fca0381d400] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7fca0381d400] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x7fca0381d400] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7fca0381d400] Format yuv420p chosen by get_format().
[h264 @ 0x7fca0381d400] Reinit context to 176x144, pix_fmt: yuv420p
[h264 @ 0x7fca03808a00] decoding for stream 0 failed
[h264 @ 0x7fca03808a00] stream 0: start_time: -7686143364045.646 duration: -7686143364045.646
[h264 @ 0x7fca03808a00] format: start_time: -9223372036854.775 duration: -9223372036854.775 bitrate=0 kb/s
[h264 @ 0x7fca03808a00] After avformat_find_stream_info() pos: 1502 bytes read:1502 seeks:0 frames:1
Input #0, h264, from 'sei2.264':
Duration: N/A, bitrate: N/A
Stream #0:0, 1, 1/1200000: Video: h264 (Constrained Baseline), 1 reference frame, yuv420p(progressive, left), 176x144, 0/1, 25 tbr, 1200k tbn, 50 tbc
[h264 @ 0x7fca0380dc00] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7fca0380dc00] nal_unit_type: 8(PPS), nal_ref_idc: 3
[AVIOContext @ 0x7fca02801b00] Statistics: 1502 bytes read, 0 seeks
My full RBSP with SPS, PPS, SEI and IDR follow:
unsigned char rbsp[1502] = {
// Offset 0x00000000 to 0x00001501
0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0xc0, 0x29, 0x8d, 0x68, 0x2c, 0x4e,
0x80, 0x78, 0x44, 0x23, 0x50, 0x00, 0x00, 0x00, 0x01, 0x68, 0xce, 0x01,
0xa8, 0x35, 0xc8, 0x00, 0x00, 0x00, 0x01, 0x06, 0x2f, 0x03, 0x08, 0x00,
0x08, 0x80, 0x00, 0x00, 0x00, 0x01, 0x65, 0xb8, 0x00, 0x04, 0x05, 0x9f,
0xff, 0xff, 0x04, 0x51, 0x40, 0x00, 0x41, 0x63, 0xc7, 0x31, 0xcf, 0xff,
0xff, 0x27, 0xff, 0xfe, 0x4f, 0xff, 0xfc, 0x9f, 0xff, 0xf9, 0x3f, 0xff,
0xf2, 0x7f, 0xff, 0xe4, 0xff, 0xff, 0xc9, 0xff, 0xff, 0x93, 0xff, 0xff,
0x27, 0xff, 0xfe, 0x4f, 0xff, 0xfc, 0x9f, 0xff, 0xf9, 0x3f, 0xff, 0xf2,
0x7f, 0xff, 0xe4, 0xff, 0xfc, 0x43, 0x05, 0xd0, 0x03, 0x2b, 0x2d, 0x11,
0x03, 0x16, 0x84, 0xb4, 0x5e, 0xc2, 0x00, 0x03, 0xa0, 0x20, 0xc7, 0x42,
0xb6, 0xb8, 0x00, 0x57, 0x66, 0x66, 0x09, 0xdc, 0xc1, 0x92, 0x50, 0x86,
0x38, 0x04, 0x8f, 0x6d, 0x83, 0xff, 0xff, 0x41, 0x5e, 0xef, 0xbb, 0xff,
0xff, 0xd0, 0x57, 0xbb, 0xee, 0xff, 0xff, 0xf4, 0x15, 0xee, 0xfb, 0xbf,
0xff, 0xfd, 0x05, 0x7b, 0xbe, 0xef, 0xff, 0xff, 0x41, 0x5e, 0xef, 0xbb,
0xff, 0xff, 0xd0, 0x57, 0xbb, 0xc0, 0x25, 0x55, 0x9a, 0xdf, 0x9f, 0x22,
0x63, 0xff, 0xff, 0x27, 0xff, 0xfe, 0x4f, 0xff, 0xfc, 0x9f, 0xff, 0xf9,
0x3f, 0xff, 0xec, 0x86, 0xfc, 0x7d, 0x2d, 0x2f, 0xf8, 0x07, 0xfa, 0xf8,
0x00, 0x37, 0x66, 0x35, 0x11, 0x29, 0x61, 0x8b, 0x30, 0x53, 0x41, 0x3c,
0xb3, 0xee, 0x59, 0xa7, 0xb7, 0xc2, 0x00, 0x03, 0x00, 0x07, 0x05, 0x86,
0x02, 0x10, 0x88, 0x38, 0x00, 0x53, 0x68, 0x93, 0x1b, 0xf4, 0x22, 0x3b,
0x90, 0x00, 0x08, 0x30, 0xd9, 0x64, 0x00, 0x02, 0x0c, 0x36, 0xe6, 0x10,
0xef, 0xa0, 0xf6, 0x52, 0xc3, 0xd9, 0x5c, 0xf8, 0x1f, 0xff, 0xb0, 0xdf,
0x80, 0x06, 0xc6, 0xaa, 0x46, 0x19, 0xd8, 0x55, 0x96, 0x5f, 0xfb, 0x38,
0xc2, 0xf8, 0x40, 0x00, 0x64, 0x0a, 0x0a, 0x30, 0x11, 0x20, 0xd4, 0x00,
0x8c, 0x99, 0xb0, 0x6c, 0xd9, 0x4a, 0x10, 0xde, 0x0c, 0x80, 0x00, 0x83,
0x8c, 0xb9, 0x88, 0xed, 0x77, 0x3c, 0x85, 0x1d, 0xcd, 0x2f, 0xff, 0xc7,
0x9c, 0x3b, 0xc0, 0xe8, 0x22, 0xb2, 0xbe, 0x04, 0xdf, 0xdf, 0xfe, 0x42,
0x80, 0x0b, 0x83, 0x88, 0xa5, 0x81, 0xc0, 0x02, 0x05, 0x08, 0x96, 0x00,
0x43, 0x64, 0x44, 0x09, 0xdf, 0xa1, 0x03, 0x88, 0xbe, 0x0d, 0x9c, 0xb1,
0x23, 0x3a, 0xe7, 0xff, 0xfd, 0x93, 0xff, 0xff, 0x27, 0xe2, 0x1f, 0xf8,
0x2c, 0x80, 0xf2, 0x10, 0xf2, 0xdf, 0x08, 0x00, 0x44, 0x00, 0x18, 0x0e,
0x81, 0x20, 0x02, 0x24, 0x0e, 0x00, 0x10, 0x20, 0x44, 0xbf, 0x06, 0xa7,
0x2f, 0xfc, 0xbf, 0x0c, 0x01, 0x02, 0xf8, 0xc4, 0xcf, 0x07, 0xc0, 0xe0,
0x00, 0x80, 0x08, 0x0f, 0x65, 0x84, 0x04, 0x0a, 0x8a, 0x03, 0x83, 0x44,
0xc0, 0x02, 0x92, 0x22, 0x20, 0x6c, 0xd9, 0x08, 0x1c, 0xfe, 0x03, 0x80,
0x8c, 0x5c, 0xb2, 0x00, 0x46, 0x2e, 0x5e, 0x04, 0x6c, 0xbb, 0x80, 0x71,
0x08, 0xf2, 0xc8, 0x10, 0x8f, 0x2c, 0x5a, 0xf2, 0x74, 0x0d, 0xb7, 0xf8,
0xd3, 0xed, 0x6d, 0x7f, 0xfc, 0x38, 0x03, 0x0e, 0xf8, 0x00, 0xee, 0x12,
0x89, 0xef, 0x22, 0x09, 0x97, 0xd3, 0xdf, 0x30, 0xc1, 0x48, 0x01, 0x80,
0x06, 0x9b, 0x80, 0x05, 0x64, 0xcd, 0x98, 0x9b, 0x29, 0x48, 0x74, 0x22,
0x32, 0x4b, 0xe0, 0x0f, 0x06, 0xd8, 0x61, 0x77, 0xb8, 0x70, 0x11, 0x8b,
0x97, 0xff, 0xc1, 0x30, 0x06, 0x1d, 0xf0, 0x00, 0xdc, 0xa1, 0x16, 0x88,
0xd5, 0xcc, 0x49, 0x17, 0xd9, 0xe8, 0x36, 0x61, 0x05, 0x55, 0x80, 0xe0,
0x0c, 0x37, 0x00, 0x21, 0xb4, 0x4a, 0x1b, 0xf4, 0x25, 0x8a, 0x40, 0x08,
0xc5, 0xcb, 0xe0, 0x70, 0x64, 0xa2, 0x13, 0xbd, 0xc3, 0x88, 0x47, 0x97,
0xf9, 0xf8, 0x07, 0xb0, 0x5f, 0x8e, 0xcc, 0xe0, 0xf8, 0x00, 0xac, 0x2f,
0x8e, 0xe6, 0x11, 0x2f, 0x51, 0xea, 0xf0, 0x40, 0xbb, 0x30, 0x4b, 0x4b,
0x0e, 0x02, 0x31, 0x72, 0xfc, 0x38, 0x84, 0x79, 0x7f, 0xe1, 0xff, 0xb0,
0x59, 0xea, 0xcf, 0xdc, 0x66, 0x12, 0xc3, 0xd5, 0xf2, 0xbd, 0x3c, 0x76,
0x02, 0x47, 0xae, 0x3d, 0xff, 0xf0, 0x24, 0x5c, 0xb3, 0xd2, 0x26, 0x2b,
0x9f, 0xf1, 0xfe, 0xc1, 0x6c, 0xf4, 0x1e, 0xe1, 0x00, 0x00, 0x80, 0x30,
0x03, 0x07, 0x97, 0x08, 0x4c, 0x85, 0x81, 0xc0, 0x02, 0x04, 0x08, 0x97,
0xe0, 0xd4, 0xe5, 0xff, 0xff, 0xf6, 0x16, 0xdb, 0xee, 0xfd, 0x1f, 0x6b,
0x6b, 0x6b, 0x6b, 0x6b, 0xc3, 0xff, 0xf6, 0x08, 0xbe, 0x0b, 0x44, 0xaf,
0xf9, 0xff, 0xb0, 0x59, 0xe0, 0x0d, 0x5f, 0x6a, 0x38, 0xde, 0x82, 0x01,
0x20, 0xc4, 0x80, 0x78, 0x00, 0x56, 0x46, 0x66, 0x84, 0xd9, 0x4a, 0xac,
0xd2, 0x00, 0x02, 0x05, 0x08, 0xb9, 0x04, 0xdf, 0xf6, 0x95, 0x9d, 0xce,
0x01, 0x82, 0xfc, 0x34, 0x3b, 0xee, 0x47, 0xc0, 0x4f, 0xaf, 0xad, 0x6f,
0x5e, 0x10, 0x17, 0x1d, 0x82, 0xc6, 0x35, 0xa0, 0x01, 0x4d, 0x91, 0x13,
0x1b, 0xf4, 0x26, 0x57, 0xc8, 0x00, 0x04, 0x08, 0x13, 0x71, 0xc6, 0x81,
0x23, 0x5b, 0xd6, 0x51, 0x2b, 0x9f, 0xff, 0xf6, 0x1f, 0xef, 0x3b, 0xcf,
0x53, 0xd7, 0x5d, 0x75, 0xd7, 0x4f, 0xff, 0xff, 0xc1, 0x07, 0x01, 0xc0,
0x21, 0xce, 0x2e, 0x5b, 0xff, 0xff, 0xb0, 0x43, 0xe0, 0x03, 0x6d, 0x10,
0xe1, 0x0a, 0x27, 0xa4, 0xdd, 0xa1, 0xce, 0x03, 0xda, 0x72, 0xdf, 0xfe,
0x29, 0x6a, 0xa5, 0xea, 0xa5, 0x80, 0x06, 0x0b, 0x68, 0x8a, 0x66, 0x62,
0xdf, 0xfc, 0x7e, 0xc1, 0x07, 0x01, 0xe6, 0x3a, 0x88, 0x7f, 0xd0, 0xe8,
0x00, 0xfc, 0x02, 0x67, 0xb1, 0xe0, 0x73, 0xd2, 0x2e, 0x03, 0xf8, 0x00,
0xc4, 0x00, 0x6f, 0x9b, 0x15, 0x6b, 0x2d, 0xcf, 0x07, 0xc0, 0x02, 0x09,
0x6a, 0x32, 0xd8, 0xba, 0x3f, 0x0c, 0x28, 0x20, 0x28, 0x20, 0x20, 0x42,
0x00, 0x83, 0xe0, 0x70, 0x23, 0x15, 0x2c, 0x01, 0x03, 0x2a, 0x63, 0xcc,
0x42, 0x99, 0x02, 0x19, 0xee, 0x01, 0xc8, 0x46, 0x96, 0x00, 0x47, 0x6c,
0xd8, 0x04, 0xaa, 0x41, 0xbc, 0x41, 0x05, 0x32, 0x00, 0x08, 0x20, 0xab,
0x9f, 0xff, 0xf4, 0x0a, 0xb4, 0xab, 0x8d, 0x3e, 0x09, 0x25, 0xff, 0xff,
0xe8, 0x9f, 0xff, 0xf4, 0x4f, 0xff, 0xfa, 0x27, 0xff, 0xfd, 0x07, 0x30,
0x04, 0x2d, 0x20, 0x47, 0x38, 0x95, 0xab, 0xfc, 0x00, 0x6b, 0x19, 0x05,
0xce, 0x53, 0x10, 0xa6, 0xed, 0x08, 0x12, 0xc0, 0x90, 0x00, 0x26, 0x1a,
0x00, 0x21, 0xfb, 0x90, 0xfc, 0x84, 0x24, 0x10, 0xcd, 0x72, 0x00, 0x14,
0xd2, 0x35, 0x91, 0x86, 0x37, 0x5e, 0x9b, 0x20, 0x00, 0x41, 0x05, 0xdc,
0xff, 0xe0, 0x18, 0x57, 0xe0, 0x02, 0x98, 0xf7, 0x33, 0x98, 0x55, 0x3d,
0x47, 0xab, 0xdf, 0xe8, 0x30, 0x01, 0xab, 0x58, 0x48, 0x53, 0xb1, 0x8a,
0x7a, 0xbd, 0xd5, 0x8f, 0x87, 0xe0, 0x01, 0x0c, 0xa4, 0x12, 0xad, 0xcc,
0x29, 0x1a, 0xd0, 0x83, 0xd9, 0xc2, 0x9a, 0xf4, 0xd0, 0x38, 0x08, 0xa4,
0xcb, 0xf2, 0xfc, 0xb0, 0xe0, 0x9c, 0x79, 0x7f, 0xff, 0xfe, 0x10, 0x61,
0xef, 0xc0, 0x01, 0x93, 0x46, 0x83, 0x76, 0xea, 0x41, 0x8d, 0xf7, 0x4b,
0x14, 0x1c, 0x00, 0x04, 0x10, 0x20, 0x18, 0x00, 0x08, 0x14, 0xf0, 0x91,
0xcd, 0xb8, 0x08, 0xd4, 0xb7, 0xcf, 0x7f, 0xff, 0xd0, 0x43, 0xdf, 0xd0,
0x81, 0x83, 0xca, 0xc0, 0x05, 0x34, 0x63, 0x27, 0x15, 0x4d, 0x52, 0x0d,
0xda, 0x5e, 0x30, 0x5e, 0x05, 0x2a, 0x0d, 0xbb, 0x90, 0x22, 0x03, 0x7a,
0x68, 0x1d, 0x21, 0xdc, 0xd4, 0xc8, 0x91, 0x4d, 0xf3, 0xcf, 0xff, 0xfa,
0x05, 0x9c, 0x53, 0xfa, 0x45, 0xff, 0xe3, 0x4f, 0xff, 0xfe, 0x82, 0xdb,
0x7d, 0xdf, 0x1c, 0x00, 0x0d, 0xc7, 0x00, 0x01, 0x01, 0x5f, 0xff, 0xfd,
0x06, 0xae, 0xf9, 0x7d, 0x31, 0xf1, 0x08, 0xe2, 0x00, 0x00, 0x80, 0x30,
0x03, 0xea, 0x2e, 0x30, 0x92, 0x6b, 0xc8, 0x70, 0x98, 0x6b, 0x90, 0xe1,
0x30, 0xd7, 0x3f, 0xf0, 0xc0, 0x34, 0x0b, 0x60, 0x00, 0x99, 0x90, 0xc8,
0xe1, 0x98, 0xdc, 0x41, 0x23, 0xcc, 0x71, 0xf0, 0x40, 0xc6, 0x17, 0xc2,
0x80, 0x07, 0xc0, 0x81, 0x40, 0x00, 0xf8, 0x09, 0xc0, 0x27, 0xb1, 0x37,
0x72, 0x02, 0xa7, 0xd5, 0xe2, 0xc0, 0x80, 0xa6, 0x78, 0x04, 0xe9, 0x89,
0x95, 0xc8, 0x0e, 0xbf, 0x57, 0x8b, 0x00, 0x80, 0x53, 0x3f, 0xf5, 0xf0,
0xd8, 0x76, 0x00, 0x09, 0xe4, 0x38, 0xcc, 0x7e, 0x90, 0x66, 0x90, 0xb5,
0x86, 0x08, 0x50, 0x00, 0x6c, 0x99, 0xb4, 0x44, 0x8c, 0x6a, 0x89, 0xc4,
0xf8, 0x27, 0xf8, 0x41, 0x1f, 0xdc, 0x10, 0x47, 0xf7, 0x60, 0x04, 0x9d,
0x0f, 0x31, 0x5c, 0x9e, 0xaf, 0x0e, 0x09, 0x89, 0x81, 0x0d, 0x34, 0xdb,
0x56, 0x00, 0x48, 0xd4, 0x34, 0x62, 0xb9, 0xbd, 0x5e, 0x1c, 0x05, 0xd9,
0x81, 0x0d, 0x34, 0xdb, 0x57, 0xfc, 0x03, 0x0e, 0x81, 0x54, 0x20, 0x04,
0x86, 0x28, 0x40, 0x09, 0x0c, 0x5f, 0xc0, 0x15, 0x24, 0x63, 0x05, 0x38,
0xc3, 0x6d, 0x5f, 0x58, 0x02, 0xa1, 0x67, 0x70, 0x53, 0x0c, 0x36, 0xd5,
0xff, 0xf0, 0x0d, 0x87, 0x70, 0x00, 0x4c, 0xc8, 0x64, 0x70, 0xcc, 0x6e,
0x20, 0x91, 0xe6, 0x38, 0xfc, 0x61, 0x70, 0x40, 0xfc, 0x20, 0x08, 0x75,
0x94, 0x10, 0x04, 0x15, 0x66, 0x61, 0x35, 0xc3, 0xec, 0x47, 0x2f, 0xab,
0xc5, 0x81, 0x01, 0x4c, 0xe1, 0x32, 0x70, 0xd5, 0x88, 0xe7, 0xf5, 0x78,
0xb0, 0x08, 0x05, 0x33, 0xff, 0x05, 0xfd, 0x82, 0xff, 0xaf, 0x80, 0x02,
0x79, 0x0e, 0x33, 0x1f, 0xa4, 0x19, 0xa4, 0x2d, 0x70, 0x42, 0x08, 0x01,
0x3c, 0x06, 0x08, 0x01, 0x38, 0x25, 0x43, 0x82, 0x61, 0x6e, 0x00, 0x12,
0xf0, 0x9b, 0x39, 0x41, 0x51, 0xea, 0xf1, 0x00, 0x98, 0x5b, 0x90, 0xe4,
0x64, 0xb8, 0x00, 0x4a, 0xd0, 0x99, 0x1c, 0xa0, 0x55, 0x7a, 0xbc, 0x41,
0x19, 0x2e, 0x7f, 0xff, 0xd0, 0x63, 0x2e, 0xa6, 0x3c, 0x00, 0x19, 0xb4,
0x49, 0x8d, 0xfa, 0x11, 0x1d, 0xd2, 0x33, 0xe8, 0xef, 0xff, 0xfe, 0xfc,
0x07, 0x00, 0x10, 0x41, 0xaa, 0x4b, 0x7e, 0x00, 0x0d, 0xd3, 0x26, 0xc4,
0xf3, 0x15, 0x1d, 0x13, 0x9f, 0x04, 0x2c, 0x0e, 0x00, 0xc1, 0x02, 0x40,
0x07, 0x81, 0xb4, 0x70, 0x75, 0x43, 0xf0, 0x3c, 0xc5, 0x70, 0x8f, 0xff,
0xff, 0xb0, 0xc7, 0x80, 0x03, 0x36, 0x89, 0x31, 0xbf, 0xc4, 0x47, 0x7d,
0x04, 0xfb, 0xf8, 0x07, 0xff, 0xb0, 0x45, 0xcb, 0x9e, 0xc2, 0x05, 0xd9,
0x82, 0x00, 0x11, 0x58, 0xaf, 0xff, 0xfb, 0x08, 0xdf, 0xff, 0xff, 0x61,
0x1b, 0x7f, 0xff, 0xec, 0x15, 0x76, 0xff, 0xff, 0xdd, 0xff, 0xff, 0xec,
0x15, 0x44, 0x7b, 0xff, 0xcd, 0x7f, 0xff, 0xec, 0x23, 0x77, 0xff, 0xfe,
0xc8, 0x9f
}
