Image component can be masked as expected, but it doesn't work when it comes to particles.
Problem
I have tried changing stencil id to be the same as the id shown in mask component in my particle shader, still not work. After reading the source code of unity mask, I learned that there were two material generated to draw mask correctly. One writes stencil and the other recover it. Only if child object of the mask was drawn between these two material can be correctly masked.
public override Material GetModifiedMaterial(Material baseMaterial)
{
...
if (desiredStencilBit == 1)
{
var maskMaterial = StencilMaterial.Add(baseMaterial, 1, StencilOp.Replace, CompareFunction.Always, showMaskGraphic ? ColorWriteMask.All : 0);
StencilMaterial.Remove(c_MaskMaterial);
c_MaskMaterial = maskMaterial;
// this code generate material that recover the stencil
var unmaskMaterial = StencilMaterial.Add(baseMaterial, 1, StencilOp.Zero, CompareFunction.Always, 0);**
StencilMaterial.Remove(c_UnmaskMaterial);
c_UnmaskMaterial = unmaskMaterial;
graphic.canvasRenderer.popMaterialCount = 1;
graphic.canvasRenderer.SetPopMaterial(c_UnmaskMaterial, 0);
return c_MaskMaterial;
}
...
}
So the more specific question is, can I adjust the render order of particles to put it between the two stencil material, in order to make them correctly masked? It would be better if someone can figure out how the stencil material works.