4

This is the first time I'm trying to use Stencil Test but I have seen some examples using OpenGL and a few on Metal but focused on the Depth test instead. I understand the theory behind the Stencil test but I don't know how to set it up on Metal.

I want to draw irregular shapes. For the sake of simplicity lets consider the following 2D polygon:

concave shape

I want the stencil to pass where the number of overlapping triangles is odd, so that I can reach something like this, where the white area is the area to be ignored:

Area to be ignored

I'm doing the following steps in the exact order:

Setting the depthStencilPixelFormat:

mtkView.depthStencilPixelFormat = .stencil8
mtkView.clearStencil = .allZeros

Stencil attachment:

let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .stencil8, width: drawable.texture.width, height: drawable.texture.height, mipmapped: true)

textureDescriptor.textureType = .type2D
textureDescriptor.storageMode = .private
textureDescriptor.usage = [.renderTarget, .shaderRead, .shaderWrite]
mainPassStencilTexture = device.makeTexture(descriptor: textureDescriptor)

let stencilAttachment = MTLRenderPassStencilAttachmentDescriptor()

stencilAttachment.texture = mainPassStencilTexture
stencilAttachment.clearStencil = 0
stencilAttachment.loadAction = .clear
stencilAttachment.storeAction = .store
renderPassDescriptor.stencilAttachment = stencilAttachment

Stencil descriptor:

stencilDescriptor.depthCompareFunction = MTLCompareFunction.always
stencilDescriptor.isDepthWriteEnabled = true

stencilDescriptor.frontFaceStencil.stencilCompareFunction = MTLCompareFunction.equal
stencilDescriptor.frontFaceStencil.stencilFailureOperation = MTLStencilOperation.keep
stencilDescriptor.frontFaceStencil.depthFailureOperation = MTLStencilOperation.keep
stencilDescriptor.frontFaceStencil.depthStencilPassOperation = MTLStencilOperation.invert

stencilDescriptor.frontFaceStencil.readMask = 0x1
stencilDescriptor.frontFaceStencil.writeMask = 0x1
stencilDescriptor.backFaceStencil = nil

depthStencilState =  device.makeDepthStencilState(descriptor: stencilDescriptor)

and lastly, Im setting the reference value and the stencil state in the main pass:

renderEncoder.setStencilReferenceValue(0x1)
renderEncoder.setDepthStencilState(self.depthStencilState)

Am I missing something because the result I got is just like there is no stencil at all. I can see some differences when changing the settings of the depth test but nothing happens when changing the settings of the stencil ...

Any clue?

Thank you in advance

endavid
  • 1,781
  • 17
  • 42
mvcs
  • 255
  • 2
  • 11

1 Answers1

4

You're clearing the stencil texture to 0. The reference value is 1. The comparison function is "equal". So, the comparison will fail (1 does not equal 0). The operation for when the stencil comparison fails is "keep", so the stencil texture remains 0. Nothing changes for subsequent fragments.

I would expect that you'd get no rendering, although depending on the order of your vertexes and the front-face winding mode, you may be looking at the back faces of your triangles, in which case the stencil test is effectively disabled. If you don't otherwise care about front vs. back, just set both stencil descriptors the same way.

I think you need to do two passes: first, a stencil-only render; second, the color render governed by the stencil buffer. For the stencil only, you would make the compare function .always. This will toggle (invert) the low bit for each triangle that's drawn over a given pixel, giving you an indication of even or odd count. Because neither the compare function nor the operation involve the reference value, it doesn't matter what it is.

For the second pass, you'd set the compare function to .equal and the reference value to 1. The operations should all be .keep. Also, make sure to set the stencil attachment load action to .load (not .clear).

Ken Thomases
  • 88,520
  • 7
  • 116
  • 154
  • Hi @Ken! Sorry for my late reply... this solved my problem. The first thing, as you said correctly, was the winding mode. I was drawing using a clockwise order, which means that the triangles are back-facing. As I wasn't defining the backFace stencil that was the reason why I didn't see any difference when applying the stencil test. Second, I did the 2 pass, one for the Stencil to store the odd/even number of triangles and another to render the scene based on the stencil buffer from the first pass. Is there any way to do it in a single pass? Thank you very much – mvcs Apr 27 '17 at 10:17
  • 1
    You're welcome. I don't think there's any way to do it in a single pass. As any given triangle is drawn over a given pixel, it's not known yet whether there will be more and whether there will ultimately be an even or odd number. So, there's no way to decide whether to actually draw the current fragment to the color attachment or to discard it. – Ken Thomases Apr 27 '17 at 15:39