What you have in your posted code isn't the correct way of applying the MODELVIEW matrix. OpenGL uses column major order for matrices. Your calculation would need to look like this:
newX=cam[0]*tx+cam[4]*ty+cam[8]*tz+cam[12];
newY=cam[1]*tx+cam[5]*ty+cam[9]*tz+cam[13];
newZ=cam[2]*tx+cam[6]*ty+cam[10]*tz+cam[14];
For the inverted matrix, you can just use a generic matrix inversion algorithm, as @datenwolf already suggested. If you can make certain assumptions about the matrix, there are easier and more efficient methods. A typical case is that you know that your transformation matrix was composed of only rotations, translations, and scaling. In that case, you can pick the matrix apart into these components, invert each, and reassemble them into the inverted matrix. Since rotations, translations and scaling are all trivial to invert, that can be done with very simple math.