How to synchronize OpenGL drawing with UIKit updates

Question

In our app we have UIScrollView above CAEAGLLayer. UIScrollView contains some UIViews (red rectangles). In CAEAGLLayer we draw white rectangles. Centers of white rectangles are the same as the centers of red rectangles. When UIScrollView scrolls, we update positions of white rectangles in CAEAGLLayer and render them.

We are getting the expected result: centers of white rectangles are always the same as centers of red rectangles.

But we can't synchronize updates of the CAEAGLLayer with the movement of the views inside UIScrollView. We have some kind of mistiming – red rectangles lag behind white rectangles.

Speaking roughly, we really want to make CAEAGLLayer lag together with the UIScollView.

We have prepared sample code. Run it on the device and scroll, and you will see that white rectangles (drawn by OpenGL) are moving faster than red ones (regular UIViews). OpenGL is being updated within scrollViewDidScroll: delegate call.

https://www.dropbox.com/s/kzybsyu10825ikw/ios-opengl-scrollview-test.zip

It behaves the same even in iOS Simulator, just take a look at the video: http://www.youtube.com/watch?v=1T9hsAVrEXw

Red = UIKit, White = OpenGL

Code is:

- (void)scrollViewDidScroll:(UIScrollView *)aScrollView {
    // reuses red squares that are gone outside thw bounds
    [overlayView updateVisibleRect:CGRectMake(...)];
    // draws white squares using OpenGL under the red squares
    [openGlView updateVisibleRect:CGRectMake(...)];
}

Edit:

The same issue can easily be demonstrated in a much simplified sample. The working xcodeproj can be find at:

https://www.dropbox.com/s/vznzccibhlf8bon/simple_sample.zip

The sample project basically draws and animates a set of WHITE squares in OpenGL and does the same for a RED set of UIViews. The lagging can easily be seen between the red and white squares.

Answer 1

In iOS 9 CAEAGLLayer has presentsWithTransaction property that synchronizes the two.

Answer 2

In fact, you can't synchronize them using current APIs. MobileMaps.app and Apple Map Kit use private property on CAEAGLLayer asynchronous to workaround this issue.

Here is related radar: http://openradar.appspot.com/radar?id=3118401

Answer 3

After digging around a little I'd like to extend my previous answer:

Your OpenGL rendering is immediately done from within the scrollViewDidScroll: method while the UIKit drawing is performed later, during normal CATransaction updates from the runloop.

To synchronize UIKit updates with the OpenGL rendering just enclose both in one explicit transaction and flush it to force UIKit to commit the changes to backbaordd immediately:

- (void)scrollViewDidScroll:(UIScrollView *)aScrollView {
    [CATransaction begin];
    [overlayView updateVisibleRect:CGRectMake(...)];
    [openGlView updateVisibleRect:CGRectMake(...)];
    [CATransaction flush]; // trigger a UIKit and Core Animation graphics update
    [CATransaction commit];
}

Answer 4

In lack of a proper answer I'd like to share my thoughts:

There are two ways of drawing involved: Core Animation (UIKit) and OpenGL. In the end, all drawing is done by OpenGL but the Core Animation part is rendered in backboardd (or Springboard.app, before iOS 6) which serves as a kind of window server.

To make this work your app's process serializes the layer hierarchy and changes to its properties and passes the data over to backboardd which in turn renders and composes the layers and makes the result visible.

When mixing OpenGL with UIKit, the CAEAGLLayer's rendering (which is done in your app) has to be composited with the rest of the Core Animation layers. I'm guessing that the render buffer used by the CAEAGLLayer is somehow shared with backboardd to have a fast way of transferring your application's rendering. This mechanism does not necessarily has to be synchronized with the updates from Core Animation.

To solve your issue it would be necessary to find a way of synchronizing the OpenGL rendering with the Core Animation layer serialization and transfer to backboardd. Sorry for not being able to present a solution but I hope these thoughts and guesses help you to understand the reason for the problem.

Answer 5

I am trying to solve exactly same issue. I tried all method described above but nothing was acceptable.

Ideally, this can be solved by accessing Core Animation's final compositing GL context. But we can't access the private API.

One another way is transferring GL result to CA. I tried this by reading frame-buffer pixels to dispatch to CALayer which was too slow. About over 200MB/sec data should be transferred. Now I am trying to optimize this. Because this is the last way that I can try.

Update

Still heavy, but reading frame-buffer and setting it to a CALayer.content is working well and shows acceptable performance on my iPhone 4S. Anyway over 70% of CPU load is used only for this reading and setting operation. Especially glReadPixels I this most of them are just waiting time for reading data from GPU memory.

Update2

I gave up last method. It's cost is almost purely pixel transferring and still too slow. I decided to draw every dynamic objects in GL. Only some static popup views will be drawn with CA.

Answer 6

In order to synchronize UIKit and OpenGL drawing, you should try to render where Core Animation expects you to draw, i.e. something like this:

- (void)displayLayer:(CALayer *)layer
{
    [self drawFrame];
}

- (void)updateVisibleRect:(CGRect)rect {
    _visibleRect = rect;
    [self.layer setNeedsDisplay];
}