I know the angle at point A and that the circle goes through point A and point B. There should be a unique solution that gives me the circle center (C) and radius (R) from this information. I've tried to find a formula as follows.
R^2 = (Bx - Cx)^2 + (By - Cy)^2
Cx = Ax - R*dy
Cy = Ay + R*dx
(dx,dy) is a unit vector for the tangent to the circle at point A, which can be found from the angle at point A with sin,cos. The center of the circle is distance R from point A in the direction perpendicular to (dx,dy).
Putting this together gives me
R^2 = (Bx - Ax + R*dy)^2 + (By - Ay - R*dx)^2
Multiplying this out gives me a quadratic for R, but the denominator of the quadratic (the /2a part) is
dx^2 + dy^2 - 1
Since (dx,dy) is a unit vector, the denominator is always 0 and I get a divide by zero error. Where have I gone wrong?