Best path from AVPlayerItemVideoOutput to openGL Texture

Question

Been pulling my hair out trying to figure out the current best path from AVFoundation videos to an openGLTexture, most of what I find is related to iOS, and I can't seem to make it work well in OSX.

First of all, this is how I set up the videoOutput:

NSDictionary *pbOptions = [NSDictionary dictionaryWithObjectsAndKeys:
                          [NSNumber numberWithInt:kCVPixelFormatType_422YpCbCr8], kCVPixelBufferPixelFormatTypeKey,
                          [NSDictionary dictionary], kCVPixelBufferIOSurfacePropertiesKey,
                                   nil];
        self.playeroutput   = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:pbOptions];
        self.playeroutput.suppressesPlayerRendering = YES;

I'm attempting three different solutions, of which only one seems to work consistently, but not sure it's the fastest. One works for a little, then breaks down with frames jumping all over the place, and one just produces black.

First off, working solution using glTexImage2D

- (BOOL) renderWithCVPixelBufferForTime: (NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        if (_cvPixelBufferRef) {
            CVPixelBufferUnlockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
            CVPixelBufferRelease(_cvPixelBufferRef);
        }
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];

        CVPixelBufferLockBaseAddress(_cvPixelBufferRef, kCVPixelBufferLock_ReadOnly);
        GLsizei       texWidth    = CVPixelBufferGetWidth(_cvPixelBufferRef);
        GLsizei       texHeight   = CVPixelBufferGetHeight(_cvPixelBufferRef);
        GLvoid *baseAddress = CVPixelBufferGetBaseAddress(_cvPixelBufferRef);


        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
        glTexParameteri(GL_TEXTURE_RECTANGLE_ARB, GL_TEXTURE_STORAGE_HINT_APPLE , GL_STORAGE_CACHED_APPLE);
        glTexImage2D(GL_TEXTURE_RECTANGLE_ARB, 0, GL_RGB, texWidth, texHeight, 0, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, baseAddress);

        glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
    } 
    return YES;
}

This method spends most of it's time Locking the base address of the pixel buffer, but the docs say it's not required if accessing data from the GPU and can impair performance. I could not figure out a way to get a texture without locking.

Next up, the almost working solution using iOSurface, this works for a bit then gets really glitchy, as if ioSurfaces are being used from previous frames:

- (BOOL) renderWithIOSurfaceForTime:(NSTimeInterval) time {
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {

         CVPixelBufferRef pb = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
         IOSurfaceRef newSurface = CVPixelBufferGetIOSurface(pb);
         if (_surfaceRef != newSurface) {
            IOSurfaceDecrementUseCount(_surfaceRef);
            _surfaceRef = newSurface;
            IOSurfaceIncrementUseCount(_surfaceRef);
            GLsizei       texWidth = (int) IOSurfaceGetWidth(_surfaceRef);
            GLsizei       texHeight= (int) IOSurfaceGetHeight(_surfaceRef);
            size_t        rowbytes = CVPixelBufferGetBytesPerRow(_cvPixelBufferRef);

            glBindTexture(GL_TEXTURE_RECTANGLE_ARB, self.textureName);
            CGLTexImageIOSurface2D(cgl_ctx, GL_TEXTURE_RECTANGLE_ARB, GL_RGB8, texWidth, texHeight, GL_YCBCR_422_APPLE, GL_UNSIGNED_SHORT_8_8_APPLE, _surfaceRef, 0);
            glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
        }
        CVPixelBufferRelease(pb);   
    }
    return YES;
}

This seems to be the best solution, if it would work. I have another process that creates textures from ioSurfaces and it works just fine, while being extremely fast too.

Finally the one that seems recommended for iOS is to use a CVOpenGLTextureCache, implementation in osx seems slightly different, and I could not get it to render anything but black, plus it seemed even slower than the first solution....

- (BOOL) renderByCVOpenGLTextureCacheForTime:(NSTimeInterval) time
{
    CMTime vTime = [self.playeroutput itemTimeForHostTime:CACurrentMediaTime()];

    if ([self.playeroutput hasNewPixelBufferForItemTime:vTime]) {
        _cvPixelBufferRef = [self.playeroutput copyPixelBufferForItemTime:vTime itemTimeForDisplay:NULL];
        if (!_textureCacheRef) {
            CVReturn error = CVOpenGLTextureCacheCreate(kCFAllocatorDefault, NULL, cgl_ctx, CGLGetPixelFormat(cgl_ctx), NULL, &_textureCacheRef);
            if (error) {
                NSLog(@"Texture cache create failed");
            }
        }

        CVReturn error = CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCacheRef, _cvPixelBufferRef, NULL, &_textureRef);
        if (error) {
            NSLog(@"Failed to copy video texture");
        }


        CVPixelBufferRelease(_cvPixelBufferRef);

        _textureName = CVOpenGLTextureGetName(_textureRef);

    }
    return YES;
}

Probably I'm not setting things up right, there's zero documentation for the texture cache in osx.

I've found it best to retain the cvpixelbufferref between render cycles, as I understand it, the texture upload can run asynchronously with CGLTexImage2d, I'm quite happy with that, several other objects may be rendered at the same time, a cglflushDrawable is eventually called when textures are eventually drawn.

Most of the apple examples I find for video to openGL Textures relate to iOS, and split the texture in two to recombine in a shader, like in this example https://developer.apple.com/library/ios/samplecode/GLCameraRipple/Listings/GLCameraRipple_main_m.html#//apple_ref/doc/uid/DTS40011222-GLCameraRipple_main_m-DontLinkElementID_11 I couldn't adapt the code directly as the texture cache has different implementations in iOS.

So any pointers would be great, it seems like vital functionality, but info I find regarding av foundation and opengl on osx seems very negative.

Update: Updated ioSurface code with use counts, works slightly longer, but still glitches out eventually.

martinjbaker · Accepted Answer · 2014-09-18T10:17:10.423

I'm starting on the same journey and know as much about OpenGL as I do about sheep farming, but did notice that your pbOptions doesn't contain kCVPixelBufferOpenGLCompatibilityKey

NSDictionary *pbOptions = [NSDictionary dictionaryWithObjectsAndKeys:
    [NSNumber numberWithInt:kCVPixelFormatType_422YpCbCr8], kCVPixelBufferPixelFormatTypeKey,
    [NSDictionary dictionary], kCVPixelBufferIOSurfacePropertiesKey,
    [NSNumber numberWithBool:YES], kCVPixelBufferOpenGLCompatibilityKey, nil];

I'm requesting the pixel buffer as kCVPixelFormatType_32BGRA rather than component and this works for me with local variables for _currentSurface (IOSurfaceRef), textureName (GLuint), _sourceWidth (int) and _sourceHeight (int)

IOSurfaceRef newSurface = CVPixelBufferGetIOSurface(pixelBuffer);
if (_currentSurface != newSurface) {
    CGLContextObj  cgl_ctx = (CGLContextObj)[[self openGLContext] CGLContextObj];
    [[self openGLContext] makeCurrentContext];

    IOSurfaceDecrementUseCount(_currentSurface);
    _currentSurface = newSurface;
    IOSurfaceIncrementUseCount(_currentSurface);
    GLsizei texWidth = (int) IOSurfaceGetWidth(_currentSurface);
    GLsizei texHeight = (int) IOSurfaceGetHeight(_currentSurface);

    if (_sourceWidth == 0 && _sourceHeight == 0) {
        // used during drawing of texture
        _sourceWidth = texWidth;
        _sourceHeight = texHeight;
    }

    if (!_textureName) {
        GLuint name;
        glGenTextures(1, &name);
        _textureName = name;
    }

    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, _textureName);
    CGLTexImageIOSurface2D(cgl_ctx, GL_TEXTURE_RECTANGLE_ARB, GL_RGBA, texWidth, texHeight, GL_BGRA, GL_UNSIGNED_INT_8_8_8_8_REV, self.currentSurface, 0);        
    glBindTexture(GL_TEXTURE_RECTANGLE_ARB, 0);
}

Thanks, I tried it out, but couldn't find any immediate benefit with the examples above. I'll be coming back to this in the next week as I tidy up my video code and play around with things a bit. — George Brown, Sep 15 '14 at 10:31
Just saw your edit' i'll give your suggestions a try soon, thanks! — George Brown, Oct 15 '14 at 21:20
Just tried it and it works for me too. Seems changing the pixel format away from YCBR fixed the glitching I was getting. — George Brown, Nov 03 '14 at 09:38

C0C0AL0C0 · Answer 2 · 2017-08-24T10:26:11.203

It appears that the corresponding method of fast video texturing on OSX is to use BiPlanar IOSurfaces where surfacePlane[0] is the luma (Y) and surfacePlane[1] is the subsampled chroma (UV). My code runs on Core Profile, so the GLenum constants to CGLTexImageIOSurface2D reflect the same. Pretty sure that only rectangle textures are supported. I use a GLSL shader to combine them and it's working great on Sierra. Briefly summarized:

NSDictionary* pixelBufferAttributesIOSurfaceBiPlanar = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithUnsignedInt:kPixelFormatTypeGL], ARC_BRIDGE(id)kCVPixelBufferPixelFormatTypeKey,
[NSNumber numberWithBool:YES], ARC_BRIDGE(id)kCVPixelBufferOpenGLCompatibilityKey,
[NSNumber numberWithBool:YES], ARC_BRIDGE(id)kCVPixelBufferIOSurfaceOpenGLTextureCompatibilityKey,
[NSDictionary dictionary], ARC_BRIDGE(id)kCVPixelBufferIOSurfacePropertiesKey,
nil];

/* luma texture */
CGLTexImageIOSurface2D(glContext,GL_TEXTURE_RECTANGLE,GL_R8,(GLsizei)textureSize.width,(GLsizei)textureSize.height,GL_RED,GL_UNSIGNED_BYTE,surfaceRef,0);
/* chroma texture */
CGLTexImageIOSurface2D(glContext,GL_TEXTURE_RECTANGLE,GL_RG8,(GLsizei)textureSize.width,(GLsizei)textureSize.height,GL_RG,GL_UNSIGNED_BYTE,surfaceRef,1);

GLSL

uniform sampler2DRect textureSampler0;
uniform sampler2DRect textureSampler1;
// ...
vec3 YCrCb;
vec2 lumaCoords = texCoord0;
vec2 chromaCoords = lumaCoords*vec2(0.5,0.5);
vec2 chroma = texture(textureSampler1,chromaCoords).xy;
float luma = texture(textureSampler0,lumaCoords).x;
YCrCb.x = (luma-(16.0/255.0)); // video range
YCrCb.yz = (chroma-vec2(0.5,0.5));
vec4 rgbA = vec4(colorConversionMatrix * YCrCb,1.0);

The color conversion matrix should be generated from the CVPixelBufferRef

CFTypeRef colorAttachments = CVBufferGetAttachment(pixelBuffer, kCVImageBufferYCbCrMatrixKey, NULL);

IOSurfaceRef is a CFTypeRef derived object and I use CFRetain/CFRelease to hold onto the surface until I don't need the texture. If CGLTexImageIOSurface2D does a GPU blit to upload the texture data, probably only need to CFRetain/Release around the call to CGLTexImageIOSurface2D.

I started with the Apple iOS sample code AVBasicVideoOutput. I ported it to Metal for both iOS & OSX in two days and then spent a week trying to figure out the OpenGL Core Profile version. If you can, use Metal: it's faster and the code is almost exactly the same on both iOS/OSX. Also, there is some good information in WWDC13 Session 507 What's New in OpenGL for OS X. In particular, that there is a GLSL shader compatibility flag that allows EAGL shaders to run mostly unmodified on OSX.

score 0 · Answer 3 · answered Apr 19 '16 at 05:56

0

When using option #2, did you set the "retains backing" property of the EAGLLayer to NO? That may be why it appears to be using the same frame in different cycles. It would be beneficial to see how you configured, or whether you configured, the layer appropriately for the view to which you're rendering framebuffers...

answered Apr 19 '16 at 05:56

James Bush

1,485
14
19

I'm not using layers at all in this app, the video frames are uploaded to openGL and processed there. For my iOS version I managed to get the CVOpenGLTextureCache working with little trouble. – George Brown Apr 19 '16 at 13:32
You are using layers; they are an integral part of any UIView or GLKView. Moreover, your textures are rendered via context to a layer. What you are not doing is configuring the layer of the view to which you're rendering, and whose context renders to that view's layer. – James Bush Apr 21 '16 at 16:57
This was originally an OSX question, I'm actually rendering to my own subclass of the NSOpenGLView, although it may go through additional processing in an FBO through shared contexts. I used very similar code to capture the display and render it to the exact same view at the same time and that would run smoothly (but video would still glitch). – George Brown Apr 21 '16 at 18:32
Is it still an OS X question? – James Bush Apr 21 '16 at 22:13
I guess it is, you can't access IOSurfaces in iOS without using private APIs, so the IOSurface solution wouldn't fly if you wanted your app in the store. For iOS, I found the CVOpenGLESTextureCache to work well (coded similar to above) and I didn't get the same problems as on OSX. I suspect I didn't configure some aspect of it correctly for OSX. – George Brown Apr 22 '16 at 13:02
You can use IOSurfaces in your app, and option number #2—as far as I can tell—needs only one fix: initializing the Core Video pixel buffer as an unsafe, mutable pointer object. Also, you need to create the pixel buffer before assigning it a value. When creating it, you can use the relevant IOSurface key in the properties dictionary; the rest, you have correct, save the absence of the CAEAGLLayer backing, which you have already been made aware of. See slides 107-108 at http://devstreaming.apple.com/videos/wwdc/2015/510jiccqsz/510/510_whats_new_in_core_image.pdf?dl=1 – James Bush Apr 23 '16 at 11:19
It's worth noting that before iOS 11, IOSurface was only available to be used indirectly in AV Foundation and Core Image (perhaps others) by specifying some parameters. In iOS 11, it appears they've fully exposed the actual IOSurface framework, so that opens up some more interesting possibilities when working with cross-media framework pixel buffer processing. – CIFilter Jun 07 '17 at 01:15
Like what possibilities? – James Bush Jun 07 '17 at 01:16
Interoperability between AV Foundation, Core Image, Metal, GLES, etc. There are new APIs exposed for these various frameworks to pass in `IOSurface` refs, so this should open up a few more pathways for direct buffer processing without needing to copy or otherwise waste time/memory transferring pixel data from one framework to another. – CIFilter Jun 07 '17 at 01:22

Best path from AVPlayerItemVideoOutput to openGL Texture

3 Answers3

Linked