If you want to check for an unrotated rectangle (i.e., two vertical and two horizontal lines), then the co-ordinates will need to be of the form
(a,x)
(a,y)
(b,x)
(b,y)
(If you want to allow for rotations then you should look up scalar products: you'll need to check whether the lines have 90 degree angles between them.)
You also mention triangles. I'm not sure if that was intentional, or if you meant the whole question to be about rectangles.
Any three points will form a triangle, as long as they're not collinear (all in a line). If you want to know whether four points all lie on the edges of the same triangle, then you'll need to check that you have three points that are collinear and one that's not on that same line. That will mean that three of the points form the vertices of your triangle, and the fourth is somewhere on one of the edges.
(I can't think of another interpretation of your question with respect to triangles. Please clarify if that's not what you meant.)