Non-monotonic depth buffer; cyclical overlapping in OpenGL

Question

Is it possible to implement custom depth buffer values so that I can implement a non-transitive ordering? E.g.

Red > Green
Green > Blue
Blue > Red

Which I imagine would be implemented in the fragment shader or something, only writing the pixel if it's on top of the existing one as per the schema above. Note that since it's non-transitive it's impossible to assign a single numeric value to each of the colors and retain the ordering.

Put another way, I would like a custom z-buffer where each element is of a non-transitive type. So the z-buffer would be a large sheet full of Red, Green or Blue values. Assign a number to each and then compare as per the schema above to determine which one is on top. If all three colors are on the same fragment then I don't care which one is on top.

Image below illustrates what I mean (this is what I want it to be able to do):

(image from Wikipedia)

Is what I want here possible in OpenGL or am I going too much against the grain? If it is possible, how severe would the reduction in performance be?

@KaiBurjack Thank you for the suggestion, but I don't just want to satisfy the contrived example I gave. Generalizing your solution to an arbitrary arrangement of colored objects is impossible I believe. I genuinely need a non-transitive z-ordering — Patrick, Feb 21 '21 at 11:47
@KaiBurjack I did give an example... the point is you can't just solve the example and then state that you've solved the entire generalized problem. I want a custom z-buffer. I'll try to make this clearer in the question — Patrick, Feb 21 '21 at 12:10
Wait, so what you want is simply making fragments of a certain colour cover fragments of another colour, without regard for depth values? — John, Feb 21 '21 at 13:07
"*this is what I want it to be able to do*" What you want doesn't make sense. What happens if all three overlap? Which color should it be? And how does any of this count as a "depth buffer", since there is no "depth" involved? — Nicol Bolas, Feb 21 '21 at 14:26
@NicolBolas If all three overlap then I don't care which one is on top, I'll update the question to mention that. — Patrick, Feb 21 '21 at 20:17
@John You could put it that way, but I would say that it's now just a different kind of depth value. — Patrick, Feb 21 '21 at 20:18
I think a z-buffer is really the wrong tool, since it's exactly this situation that it's supposed to prevent. Custom fragment shader? — Neil, Feb 21 '21 at 21:26
I would say this lacks detail to attract a good answer (what are you actually trying to accomplish? What have you tried and how did it fail?). My guess would be to use a UAV or compute shader — user2407038, Feb 21 '21 at 22:01
@Patrick: "*I would say that it's now just a different kind of depth value.*" You can call it whatever you want, but that doesn't make it anything like what the word "depth" represents. — Nicol Bolas, Feb 21 '21 at 22:35
@NicolBolas Well I was just thinking of it as a more abstract depth, e.g. one represented by a vector rather than a scalar. I don't think this quibbling on definitions is very useful though — Patrick, Feb 21 '21 at 23:31
@Neil That's certainly the sort of solution I would like. Unfortunately I can't really have an readable and writable frame buffer accessible in the fragment shader so I'm not sure how to do it with the fragment shader — Patrick, Feb 24 '21 at 00:38

Stuntddude · Answer 1 · 2021-02-22T12:05:34.363

What you want to do is not only possible, it's relatively fast and simple. I've implemented something like this in my game in order to render the penrose triangle.

The trick is to use a stencil buffer instead of the depth buffer. There are actually several different ways you can use the stencil buffer to achieve a circular ordering effect, depending on your exact needs. I'll describe one possible algorithm here:

Draw the red layer with no stencil test, while writing 1s to the stencil buffer.
Draw the green layer with stencil test not equal to 1, while writing 2s to the stencil buffer. This will prevent it from being drawn where red has already been drawn, effectively putting green "behind" red.
Draw the blue layer with stencil test not equal to 2. This will allow it to be drawn on top of red, but not green, effectively placing it between them.

The same technique generalizes to N layers like so:

- for each layer X going front-to-back, from `N - 1` down to `0`
    - draw layer X with stencil test `not equal to X + 1`, while writing `X` to the stencil buffer

Code for this might look something like:

glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
for (int X = N - 1; X >= 0; --X) {
    glStencilFunc(GL_NOTEQUAL, X + 1, 0xFF);
    // drawLayer(X);
}

This algorithm only guarantees that layer X will correctly interact with layers (X - 1) % N and (X + 1) % N. If you want more sophisticated rules, you will need a more sophisticated algorithm.

Note that with an 8-bit stencil buffer, you will be limited to 255 layers.

Also note that because this algorithm doesn't use the depth buffer, it works for 3D scenes as well - just enable the depth test like normal, and you're off to the races. That's how I'm able to render the penrose triangle example in 3D.

Thank you! The stencil buffer looks perfect (I'm kind of a noob with OpenGL so I wasn't aware of it). I will try to remember to mark this as answer once I've given this method a try, — Patrick, Feb 22 '21 at 12:25
I was thinking of writing to a framebuffer, then using it as a texture where you'd draw everything, then check for colour values on each pixel and only draw the relevant ones. But this solution is way simpler. — John, Feb 22 '21 at 18:50
The stencil buffer was designed for just this sort of thing, so I expect this to be fast, as well. Great answer. — Neil, Feb 24 '21 at 17:20

Non-monotonic depth buffer; cyclical overlapping in OpenGL

1 Answers1