I think I've got a solution. The reason links X1/2Y aren't valid is that they cross over each other when you draw the links from start to finish. That got me thinking that testing for intersection should work.
The first step to doing this is finding the "creation vector" of the new connection that's being made (the dashed line). In this case, node2 is the startPosition, with the endPosition being the center node. This is done as follows:
creationVector = normalize(endPosition - startPosition);
Next, you need to calculate two points related to node2 in the diagram: the incomingPoint and outgoingPoint. To maintain the clockwise direction, incomingPoint needs to be to the left of the creation vector and the outgoingPoint needs to be to the right. In the diagrams below incomingPoint is red while outgoingPoint is blue. This is done as follows:
incomingPoint = nodeAPosition + new float2(-creationVector.y, creationVector.x);
outgoingPoint = nodeAPosition + new float2(creationVector.y, -creationVector.x);
The last bit is drawing two lines and testing whether there's an intersection between them. We'll start with the invalid links, X1/2Y. In this case, the two lines we're drawing are: nodeX to incomingPoint, and outgoingPoint to nodeA. The solution I found for testing for intersection comes from this blog post. I'll include my version below for completeness:
private float2 IntersectionPointGet(
float2 a1,
float2 a2,
float2 b1,
float2 b2,
out bool isIntersection)
{
float tmp = (b2.x - b1.x) * (a2.y - a1.y) - (b2.y - b1.y) * (a2.x - a1.x);
if (tmp == 0)
{
isIntersection = false;
return float2.zero;
}
float mu = ((a1.x - b1.x) * (a2.y - a1.y) - (a1.y - b1.y) * (a2.x - a1.x)) / tmp;
isIntersection = true;
return new float2(
b1.x + (b2.x - b1.x) * mu,
b1.y + (b2.y - b1.y) * mu
);
}
As you can see, links X1/2Y intersect and thus are invalid:

Now compare this to A1/2B, which do not intersect and thus are valid:

The nice thing about this solution is that you only need to test one link pair. If X1/2Y is invalid, we can assume A1/2B is valid. That means that link AB is the one that needs to be split, while link XY should be left alone.