2

I receive an MPEG TS container over network (UDP). It contains two streams: an mpeg2video vidoe stream with yuv420p pixel format and a data stream encoded using a propitiatory KLV format.

My receiver program must be in Python. So, I can't use FFMPEG library (like AVFormat, AVCodec) directly.

Now my problem is as follows:

I need to receive video frames and save them as RGB image as raw numpy array. I also need for each frame to parse the corresponding KLV data. There is a one to one relationship between video frames and KLV data units.

I thought I use ffprobe to output the packets including their payload data from incoming container and then parse the output of ffprobe to get the images and metadata:

$ ffprobe -show_packets -show_data -print_format json udp://127.0.0.1:12345 > test_video.packets.data.json

This gives me an output (in test_video.packets.data.json file) like:

{
    "codec_type": "video",
    "stream_index": 0,
    "pts": 140400,
    "pts_time": "1.560000",
    "dts": 136800,
    "dts_time": "1.520000",
    "duration": 3600,
    "duration_time": "0.040000",
    "size": "21301",
    "pos": "3788012",
    "flags": "K_",
    "side_data_list": [
        {
            "side_data_type": "MPEGTS Stream ID"
        }
    ],
    "data": "... "
},
{
    "codec_type": "data",
    "stream_index": 1,
    "pts": 140400,
    "pts_time": "1.560000",
    "dts": 140400,
    "dts_time": "1.560000",
    "size": "850",
    "pos": "3817904",
    "flags": "K_",
    "side_data_list": [
        {
            "side_data_type": "MPEGTS Stream ID"
        }
    ],
    "data": ".... "
}

I can extract the KLV data from the data packets and parse it. However the data from the video packets in encoded as mpeg2video video with yuv420p pixel format.

My Questions:

  1. How can I get the raw pixel values from that mpeg2 encoded payload?
  2. Is it possible to use ffmpeg to receive the original container and copy it (with both streams) into a new container, but with raw video instead of mpeg2 video? if yes, how? what should be the command? I tried for example: ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -codec rawvideo -pix_fmt rgb24 -map 0:1 -codec copy -f mpegts udp://127.0.0.1:11112, but it gives me again mpeg2 encoded video data in payload of video packets
zardosht
  • 3,014
  • 2
  • 24
  • 32
  • 1
    1. The frame has to be decoded to get access to the raster. 2. MPEG-TS does not support rawvideo. MKV, NUT, or even AVI do. – Gyan Nov 25 '20 at 04:55
  • @Gyan Thanks. 1. Can you please point to a way on how I could decode the frame into a raw RGB image? 2. Can you please give a reference (link to documentation, Wikipedia, etc.)? – zardosht Nov 25 '20 at 08:49
  • 1
    1. Use a format filter in ffmpeg to convert to `rgb24` 2. Codec support in a container is not documented. You have to either inspect the source or run a command and check for error. – Gyan Nov 25 '20 at 09:58
  • The problem is if I decode the packets using ffmpeg, for example like this "ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -codec rawvideo -pix_fmt rgb24 udp://127.0.0.1:11112" I lose the packet timestamps. I need those to synchronize my video stream with KLV stream. To be concrete: I need a way to extract an RGB image and its corresponding KLV data (there is a 1-to-1 correspondence using packet PTS timestamps, for data and vidoe packets). – zardosht Nov 25 '20 at 10:13
  • 1
    That command will fail since you haven't set output muxer using `-f`. If you did, and set it to `mpegts` then packet timestamps would carry over. The problem is mpegts doesn't support rawvideo. However, since we are talking about uncompressed frames. You can just mux it a private data stream, and interpret the packet data correctly. – Gyan Nov 25 '20 at 11:01
  • Yes, actually I had set the -f mpegts for the output, but it encoded the video frames again to mpeg2. Can you please elaborate more on how to "mux to a private data stream"? I think this exactly what I need. But I am quite unfamiliar with ffmpeg. Btw, if you write it as an answer I can upvote and accept it :) – zardosht Nov 25 '20 at 16:07
  • Simply `ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -c:v rawvideo -pix_fmt rgb24 -f mpegts udp://127.0.0.1:11112`. It will automatically be muxed as a private data stream. – Gyan Nov 25 '20 at 16:12
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/225109/discussion-between-zardosht-and-gyan). – zardosht Nov 25 '20 at 16:19
  • But I also need the data stream. As said, I had actually tried the following command: "ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -codec rawvideo -pix_fmt rgb24 -map 0:1 -codec copy -f mpegts udp://127.0.0.1:11112", but it gives me encoded video data in payload of vidoe packets and I don't know how to decode them to RGB. – zardosht Nov 25 '20 at 16:22
  • 1
    `ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -map 0:1 -c copy -c:v rawvideo -pix_fmt rgb24 -f mpegts udp://127.0.0.1:11112` – Gyan Nov 26 '20 at 04:48
  • Thanks :). This solved the problem. If you write it as an answer I can accept it. – zardosht Nov 26 '20 at 15:20

1 Answers1

5

MPEG-TS supports a limited number of video codecs. However, ffmpeg's muxer will silently mux even unsupported streams as private data streams.

To mux a raw RGB stream, convert to rgb24 pixel format and code using rawvideo codec.

ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -map 0:1 -c copy -c:v rawvideo -pix_fmt rgb24 -f mpegts udp://127.0.0.1:11112

Gyan
  • 85,394
  • 9
  • 169
  • 201