I've been wrestling with this for days. I think I've finally narrowed it down to a problem with the per vertex tangents, but I'm not sure the best way to fix it.
Context is iPhone app, opengl es2 using my own engine. My shader is a bump map (normal map) variety using the supplied per vertex tangent to create a TBN matrix. The vertex shader transforms the light vectors and eye vectors to tangent space, passes them to the fragment shader and calculates the lights. But some geometry in my first two test models is showing weird artifacts in the lighting. It is easiest to see in the specular component.
In trying to debug this I've replaced the normal map with a flat normal png.. all pixels are 128,128,255. I also hard coded the color.
My first model is a button shape. It shows the artifact as a scalloped specular hit on the outside ring. Important to note is that the UV mapping method here was 'flat' to make the sides which are perpendicular to the mapping basically streak the texture across there. I would think this would make the tangents hard to calculate for that geometry since two of the points would have the exact same texture coordinate.
I tried debug renders setting gl_FragColor to different variables. When setting it to the per vertex normals we see the scalloping is gone, indicating that the normals are correct. But when setting it to the per vertex tangents you can see the scallop.
The next shape is a simple sphere. With it the very top and bottom of the model is where the artifact shows. In the wireframe you'll see that this is where several triangles are meeting at one vertex. I can't wrap my head around what that means for the tangents as each triangle has completely different UV mapping there.
I suppose I'll get a lot of flack if I don't show the shader code...
Vertex Shader:
v_fragmentTexCoord0 = a_vertexTexCoord0;
gl_Position = u_modelViewProjectionMatrix * vec4(a_vertexPosition,1.0);
vec3 normal = normalize(u_normalMatrix * a_vertexNormal);
vec3 tangent = normalize(u_normalMatrix * a_vertexTangent);
vec3 bitangent = cross(normal, tangent);
mat3 TBNMatrix = mat3(tangent, bitangent, normal);
v_eyeVec = -a_vertexPosition.xyz;
v_eyeVec *= TBNMatrix;
v_lightVec = u_lightPosition - a_vertexPosition.xyz;
v_lightVec *= TBNMatrix;
v_normal = a_vertexTangent;
Fragment Shader:
vec3 norm = texture2D(u_normalSampler, v_fragmentTexCoord0).rgb * 2.0 - 1.0;
vec4 baseColor = vec4(0.6015625,0.0,0.0,1.0); // is normally texture2D(u_textureSampler,v_fragmentTexCoord0);
float dist = length(v_lightVec);
vec3 lightVector = normalize(v_lightVec);
float nxDir = max(0.0, dot(norm, lightVector));
vec4 diffuse = u_lightColorDiffuse * nxDir;
float specularPower = 0.0;
if(nxDir != 0.0)
{
vec3 cameraVector = v_eyeVec;
vec3 halfVector = normalize(v_lightVec + cameraVector);
float nxHalf = max(0.0,dot(norm, halfVector));
specularPower = pow(nxHalf, u_shininess);
}
vec4 specular = u_lightColorSpecular * specularPower;
gl_FragColor = (diffuse * vec4(baseColor.rgb,1.0)) + specular;
In trying to figure out the shader and looking at all the samples and tutorials online I could find... I was confused by why the bump map shaders wouldn't just perturb the normal map supplied by the 3d model. I guess without some sort of reference point how you know HOW to modify model's normal? By what orientation? I guess that's what the TBN matrix is for. But in my test above the normal vector in the normal map is 0,0,1 - straight up. So it seems like it shouldn't be modifying it at all. Anything times 1 is still 1. But there must be something screwed up enough in the math or in the TBN matrix that it's not coming up with the same normal as the one in the 3d model. Then that idea struck something with me.. that in my vertex shader I'm also first multiplying the vertex normal by the normalMatrix to get it in to model space. But then again, those debug renders are of the vertex normal before it was transformed.
Is there another method to perturb the model's normal without using a TBN matrix? Rendering with a phong shader without the normal map doesn't show the artifact.
UPDATE: I'm almost positive that the problem is the precalulated tangents created by the importing app. After trying different models with different UV Maps I'm finding similar looking problems, sometimes it's a darkness instead of a highlight. So unless I can apply a normal map without a TBM matrix or converting to tangent space, I will need to find another importer.
UPDATE #2: I just found this other question that sounds like it might be a clue to the real problem. 3d graphics, unit vectors and orthogonal matrices
It's important to note that these weird lights don't happen on vertices but only in between them. Even on the sphere we're seeing a ring which is between the top vertex and the ones right below it. I'm starting to believe that this is an interpolation problem. Take just one of those triangles up there:
Ignore the bad bitangent for a second cause I'm recalculating that. But the tangents have opposite polarity. So while being interpolated between those two states your going to get vectors that point all over the place.
The new question is how do I solve it? Fixing the tangents? Fixing the shader to deal with it?
