So I'll try to describe the problem in detail, and I'd like some critique on the validity and performance of the process I use to solve it. My main concern is the validity, which I cannot seem to prove.
The idea is that we have a 2D polygon in 3D space (which is described by all its hull points. Each point also has a normalvector which indicates an infinite extrusion. I apologize for using the misleading term 'normal', it's just a normalized vector that represents a normal in a broader context not relevant to this algorithm. To completely clarify, these vectors are neither perpendicular to the plane nor the same on each point. As can be seen in the following image, where the vectors are the red lines:
This means that the sides of this extrusion do not form a flat, but a skewed plane. One can look at its extrusion sides like a Skew polygon (http://en.wikipedia.org/wiki/Skew_polygon).
Anyway, I'd like to see if a point P is inside the extrusion or not. My current procedure is as follows:
- Calculate normal N=(A-B)x(B-C) where A,B,C are any three clock-wise points on the 2D polygon. This gives me the 2D polygon plane normal.
- Calculate the final plane variable D to get NxX+NyY+Nz*Z+D = 0. Do this by filling in point P in X,Y,Z. This gives the extruded plane.
- Extrude the normals at all hull points to get the new hull on the extruded plane. Simply done by finding the intersection of the line they define and the plane.
- Here comes the tricky part. Now I have a convex hull on a 3D plane where I know the point is coplanar. I have seen a lot of solutions already on stackoverflow concerning solving linear systems, etc... but they either do not consider 3D, lack a good implementation or look too expensive.
My approach is using an assumption that I think is both necessary and sufficient but I cannot prove that last part (the necessary part is obvious). I make the following claim:
For coplanar convex hull consisting of points A->Z and an additional coplanar point P. Then P is within the convex hull if and only if for all clock-wise sequential points A and B and some other third sequential point C the line C-P passes through P before passing through the line A-B.
In the example image above: A'-B' is crossed by D'-P going through P before A'-B', for B'-C' this is achieved by A', for C'-D' also by A' and for D'-A' by B'.
I implement this idea as two 3D parametric lines crossing eachother (A-B and P-C) and looking at the parameter's sign. The parameter in P-C should be non-positive because the crossing point should lie behind P for a parameter going to C.
In essence this would result in a O(n)O(n^2) algorithm with n being the #points of the polygon (which seems to beat all solutions I've seen so far). However, even though I cannot find a counterexample to the above assumption, I also cannot prove it.
Edit: I believe it is possible to prove that a sufficient condition is that for A and B, all other hull points need to satisfy the assumption (and not just C).
Edit2: Clarified claim.
