1

I am trying to identify if a video file has pixelation or not,

Currently, I have a watch folder where videos are dropped, I have a trigger every 120 seconds that check the folder to see if there are any video files. If there is a file, it uses MediaInfo command-line to extract the encoding of the video and audio. I also use ffprobe on a video to see if it throws out an error like moov atom not found to know if the video is corrupted.

However, I want to be able to tell if a video file has a pixelation. I know that I can use str, std, to print out the debug logs.

However, I am finding it difficult to find documentation on how to detect pixelation.

Do you guys know if there a way to tell that a video file may have pixelation, even through a log output of the video?

I am already running a script that reads a log and finds information on the log to tell me about packet drops in a stream, so if you know how I output such log that tells if there is pixelation, or if the debug log has certain information that tells this, please let me know.

A Person
  • 441
  • 7
  • 19
  • 1
    Please define "pixelation". Can you show an example of a video (frame) that shows pixelation, and one that does not? Since you mention packet drops, are those videos recordings of streams from unreliable transport (e.g. MPEG-TS over RTP/UDP)? – slhck Nov 14 '19 at 11:37
  • pixelation is hard to define, I would say if the video is distorted, has losses in audio, video static, good example with losing frames. – A Person Nov 14 '19 at 11:46
  • Please also answer my second question. So basically detecting if a stream has packet loss? Are you getting *direct* recordings from the streams, and how are they made? – slhck Nov 14 '19 at 11:51
  • The stream are coming in via UDP link. They are encoded live in different formats are there are more than one location where I am collecting these from, for the packet loss, I am looking at the debug log to see if the log contains the value 'PES mismatch' if it does, then it counted as a packet drop. – A Person Nov 14 '19 at 11:54

1 Answers1

3

Any lost packet will inadvertently lead to decoding errors (both in video/audio streams), so in principle you already have your solution if you inspect the debug log while decoding. Is there a mismatch or decoding error? Then your file has probably been corrupted.

The "problem" is that decoding is somewhat robust against losses, and errors from lost frames or parts of frames can be concealed by the decoder. This depends on the decoder; some to better than others here.

How to tell if a file has problems?

So, a file may only have few lost packets, and you'd consider it as faulty when looking at the bitstream only (i.e. the ffmpeg log), but it will decode just fine, and a human viewer will not be likely to notice the error. At some point, of course, a human will definitely notice these problems.

That means you're going to either need some threshold as to how many packet errors are acceptable, or a more detailed analysis on the pixel level. That means: decode the video and analyze the pixels whether you can detect errors stemming from packet/slice losses (what you are calling "pixelation"). Of course, it'd be easier if you had access to the original video to compare with, but I assume that is not the case.

You have to decide what that threshold is. Take a sample of videos and label them: are they "pixelated" or not? This is nothing that an algorithm can simply decide for you. Then match your count of packet errors against your binary decision of whether to accept or reject the video. You should hopefully see some correlation there, although, of course, it will vary a lot depending on which frame is lost or how much loss there was.

Are there any algorithms for that?

There are of course algorithms that can help estimating the quality of a decoded video if you don't have a reference. These are called "no-reference" pixel-based quality models.

Some algorithms you may want to look into:

All of those metrics are not quite good in terms of predicting human perception. It's very likely that you'll get some false positives.

An even better way to predict the quality of such videos would be to use "hybrid no-reference" algorithms which take both the bitstream and the video pixels into account. There are some standardized algorithms out there (e.g. ITU-T J.343.1), for which Rohde & Schwarz offers a commercial implementation (see page 20-23 for the issues that you are probably facing).

slhck
  • 36,575
  • 28
  • 148
  • 201