0

I call the following method to stream frames from a USB Camera to org.WebRTC's Video Capture Observer. It works fine, except for some corrupted frames (green, or distorted grey frames) that appear every couple of seconds in the remote peer's video display. I want to verify that the nv21Buffer or the videoFrame are good pictures before sending them further to the observer.

    public void addFrame(ByteBuffer frame) {
        try {
            byte[] imageArray = new byte[frame.remaining()];
            frame.get(imageArray);
            NV21Buffer nv21Buffer = new NV21Buffer(imageArray, 640, 480, () -> JniCommon.nativeFreeByteBuffer(frame));
            VideoFrame videoFrame = new VideoFrame(nv21Buffer, 0, System.nanoTime());

            // before sending the videoFrame further, I need to validate if it is a good picture / frame
            capturerObs.onFrameCaptured(videoFrame);

        } catch (
                Exception e) {
            Log.d("addFrame", e.getMessage());
        }
    }

Is there any built in tools / methods to verify the goodness of the frames before sending them further?

kostyabakay
  • 1,649
  • 2
  • 21
  • 32
WM1982
  • 53
  • 4

1 Answers1

1

Nv21 is raw pixels. In memory it is just an array of bytes. There are no checksums, or other structure that can be wrong. If it is the correct number of bytes it is “good”

Your only option is to do some form of image processing and compare it to other frames, or develop a heuristic/algorithm to detect these. But there is not one built in.

szatmary
  • 29,969
  • 8
  • 44
  • 57
  • Is this correct also in regard to these other formats: `YUV`, `RAW`, `RGBX` ? I can get them all in `ByteBuffer` format from my USB Camera. What if I produce a `Bitmap` of the frame, would that make it easier to verify the image? – WM1982 Apr 13 '20 at 15:06
  • Nope. This is true for all raw pixel formats. A bitmap is the same thing, just with a header that has the pixel format and resolution. It still doesn’t have a checksum. – szatmary Apr 13 '20 at 15:08
  • @WM1982 I have a similar YUV byte array from a camera source which I'd like to send over webRTC. Issue is the YUV image byte arrays, when put into a NV21 buffer, create a ton of distortion and noise when transmitted. Did you do any decoding/configuration of your stream? Thanks! – mger Nov 03 '21 at 16:41