3

I'm currently learning how to use Metal and having some difficulty using the stencil buffer – possibly because its the wrong solution for the problem I have.

The problem: I have a tree of 2D render nodes, quads, that I'm rendering with metal. For some quads, I'd like to enable a 'clipping mask', that clips the rendering of all its sub-nodes to within its bounds.

I imagined that this might be a good use-case for the stencil attachment (Metal is my first foray into low-level graphics APIs) but am not 100% sure.

Having set-up the depth attachment however, I've got no idea of what to actually do with it. Is it even possible to implement this idea of nested clipping masks with this method?

My rough idea for how it might work would be:

  1. Set-up a pipeline state for each quad as usual

  2. Set-up a couple of depth stencil states, one for tree elements that will clip, and one for nodes that won't. (Write masks of 0xFF and 0x00 respectively.)

  3. Begin a render pass as usual and begin traversing the tree.

  4. If a node should clip, use the clipping depth stencil state otherwise the non-clipping.

Any idea if this is the right approach?

Any thoughts as to the specifics of tackling this problem. i.e configuration of the MTLStencilDescriptor and its read/write masks and various comparison operations and functions. How I would set the stencilReferenceValue on the render command encoder? Increment it at each level of the tree?

EDIT: A Similar question attempts to tackle this problem in Open GL (although the solution comes with its own compromises), so it appears that the above problem can be tackled using a stencil attachment.

The solution in the linked question notes "by giving each level in your scene graph a higher number than the last, you can assign each level its own stencil mask," but comes with the caveat: "Of course, if two siblings at the same depth level overlap, or simply are too close, then you've got a problem."

Is there a better way of achieving this with the capabilities that Metal provides? An idea of/pointers to a recommended algorithm/method to do this within the capabilities of Metal's API would be appreciated!

Tricky
  • 7,025
  • 5
  • 33
  • 43
  • Are the 2D nodes rendered in 3D? That is, might they be arbitrarily rotated? Or are they always "flat" in the same orientation as the window into which they are being rendered? If the latter, the scissor rect may be much easier. Also, could it be that using `CALayer`s would get you what you need largely for free? – Ken Thomases Oct 15 '17 at 18:33
  • They can be arbitrarily rotated, so no scissor rect unfortunately. `CALayer`'s do indeed provide *most* of what I need – but not all! – Tricky Oct 15 '17 at 18:37
  • If they can be arbitrarily rotated, what does clipping actually mean? What are the "bounds" of a node and, if a node's child can be rotated relative to the node's plane, what is considered inside or outside of those bounds? Can you provide an image illustrating what you want? – Ken Thomases Oct 15 '17 at 20:31
  • I'm trying to follow as close a model to a CALayer tree (position and bounds) as possible. Each node is a plane (quad) that has a center (position) and (bounds). I'm not using the depth dimension currently (everything zeroed on z). I've made an edit to the question to include a link to a question that attempts to tackle this problem in Open GL. – Tricky Oct 16 '17 at 11:22

0 Answers0