I'm writing a PBR shader and I'm trying to determine the correct way to generate the alpha value that is output for the image. The goal is to have a final image that is 'premultiplied alpha', so it can be used with another Over composite operation later on (say, compositing it over another background image). This all works great, except for the case of specular highlights on transparent surfaces.
Reading this article: https://google.github.io/filament/Filament.md.html#lighting/transparencyandtranslucencylighting
In particular:
Observe a window and you will see that the diffuse reflectance is transparent. On the other hand, the brighter the specular reflectance, the less opaque the window appears. This effect can be seen in figure 63: the scene is properly reflected onto the glass surfaces but the specular highlight of the sun is bright enough to appear opaque.
They also include the code snippet:
// baseColor has already been premultiplied
vec4 shadeSurface(vec4 baseColor) {
float alpha = baseColor.a;
vec3 diffuseColor = evaluateDiffuseLighting();
vec3 specularColor = evaluateSpecularLighting();
return vec4(diffuseColor + specularColor, alpha);
}
When rendering glass the metallic level would be 0, so it's not pulling from the baseColor at all, and just using the specular level. This all makes sense and means specular highlights can still occur even on the transparent glass, as per the above quote. I don't want to just multiply by alpha at the end, since my baseColor texture would already be a premultiplied alpha image, such as something containing a decal. For example let's say I'm drawing a pane of glass with a sticker on it that's the letter 'A'.
My question is: for a pixel that has a specular highlight on a transparent portion of the glass, what should the alpha value be for compositing to work downstream? The alpha would be 0 according to the above code, which would make an Over operation later on behave poorly. I say this since a property of pre-multiplied alpha is that max(R,G,B) <= A, but this specular highlight would have max(R,G,B) > A. This would result in the over operation being additive between the specular highlight and the background it's being composited over, blowing out the final image. Should I set the alpha to max(specular, alpha), to give a more useful alpha value for these pixels? Is there a more 'correct' way to handle this? Thanks