WARNING - Almost every thing I know about using barycentric coordinates, and using matrices to solve linear equations, was learned last night because I found this question so interesting. So the following may be wrong, wrong, wrong - but some test values I have put in do seem to work.
Guys and girls, please feel free to rip this apart if I screwed up completely - but here goes.
Finding barycentric coords in 3D space (with a little help from Wikipedia)
Given:
v0 = (x0, y0, z0)
v1 = (x1, y1, z1)
v2 = (x2, y2, z2)
p = (xp, yp, zp)
Find the barycentric coordinates:
b0, b1, b2 of point p relative to the triangle defined by v0, v1 and v2
Knowing that:
xp = b0*x0 + b1*x1 + b2*x2
yp = b0*y0 + b1*y1 + b2*y2
zp = b0*z0 + b1*z1 + b2*z2
Which can be written as
[xp] [x0] [x1] [x2]
[yp] = b0*[y0] + b1*[y1] + b2*[y2]
[zp] [z0] [z1] [z2]
or
[xp] [x0 x1 x2] [b0]
[yp] = [y0 y1 y2] . [b1]
[zp] [z0 z1 z2] [b2]
re-arranged as
-1
[b0] [x0 x1 x2] [xp]
[b1] = [y0 y1 y2] . [yp]
[b2] [z0 z1 z2] [zp]
the determinant of the 3x3 matrix is:
det = x0(y1*z2 - y2*z1) + x1(y2*z0 - z2*y0) + x2(y0*z1 - y1*z0)
its adjoint is
[y1*z2-y2*z1 x2*z1-x1*z2 x1*y2-x2*y1]
[y2*z0-y0*z2 x0*z2-x2*z0 x2*y0-x0*y2]
[y0*z1-y1*z0 x1*z0-x0*z1 x0*y1-x1*y0]
giving:
[b0] [y1*z2-y2*z1 x2*z1-x1*z2 x1*y2-x2*y1] [xp]
[b1] = ( [y2*z0-y0*z2 x0*z2-x2*z0 x2*y0-x0*y2] . [yp] ) / det
[b2] [y0*z1-y1*z0 x1*z0-x0*z1 x0*y1-x1*y0] [zp]
If you need to test a number of points against the triangle, stop here. Calculate the above 3x3 matrix once for the triangle (dividing it by the determinant as well), and then dot product that result to each point to get the barycentric coords for each point.
If you are only doing it once per triangle, then here is the above multiplied out (courtesy of Maxima):
b0 = ((x1*y2-x2*y1)*zp+xp*(y1*z2-y2*z1)+yp*(x2*z1-x1*z2)) / det
b1 = ((x2*y0-x0*y2)*zp+xp*(y2*z0-y0*z2)+yp*(x0*z2-x2*z0)) / det
b2 = ((x0*y1-x1*y0)*zp+xp*(y0*z1-y1*z0)+yp*(x1*z0-x0*z1)) / det
That's quite a few additions, subtractions and multiplications - three divisions - but no sqrts or trig functions. It obviously does take longer than the pure 2D calcs, but depending on the complexity of your projection heuristics and calcs, this might end up being the fastest route.
As I mentioned - I have no idea what I'm talking about - but maybe this will work, or maybe someone else can come along and correct it.