I've spent a bit of time researching this and can't find a good solution.
My problem is finding an efficient way to create normals in a tessellation shader. I creating a mesh for a cave system procedurally then tessellating it in real time to improve resolution. I have most of the system working but I'm struggling to find a reasonable way to create the normals.
Here is a link to my paper describing the complete system for those interested:
https://cs.anu.edu.au/courses/CSPROJECTS/15S2/Reports/Tony_Oakden_Report.pdf
I use a triplanar shader to synthesize texture coordinates and I'm using a heightmap to deform the mesh in the tesselation evaluation shader. The deformation works fine but the only way I've found to reliably create normals is in the geometry shader, which works but the GS is too slow really.
I had an idea that I could use TextureGather to pick up four texels from the height map and use those to find the approximate normal at that point (get the x and y gradients, then get the edge vectors, then the cross product and normalise etc to get the normal)but I can't get it to give good results. I found a few links to this problem on the web but no specific answers.
Seems like synthesizing normals for procedurally generated mesh in the shader is a tricky problem and I'm wondering if there are known solutions to these types of problems I can look at. Any suggestions, links, techniques, blogs that people can point me too would be most appreciated.