A group element does not have a shape of its own. So you have to look at its content.
If the group element has some transformation, you might either apply this transformation to each child element, or apply the reverse operation to the current mouse position. In the latter case, you will have to apply some correction to the final result, since the distance will be in transformed coordinates so you have to revert it back to untransformed coordinates.
There are several ways how you could define the shape of a path, and each definition yields a different distance metric. Ordered from fast but crude to slow but exact, the options that I can think of are
- Axis-aligned bounding box. Simply take all points, including Bézier control points, and take the minimum and maximum for each coordinate. This defines a bounding rectangle, and you can compute the distance to that.
- Convex hull. Take all points, again including control points, and compute the convex hull of these. The path will be contained within that hull. Compute the distance to that.
- Point cloud. Take all the points, perhaps again including control points but perhaps not, and find the closest one to our input. Use the distance to that.
- Piecewise linear. Ignore control points, but consider your path as a sequence of line segments. Compute the distance to each, then take the minimum.
- Flattened path. Use De Casteljau's algorithm to approximate curved paths by piecewise linear paths until some error bound is met, then do as described above.
- Mathematically exact. Look at each curved segment, and actually compute the minimal distance. For this you need to find the points where the tangent to the curve is orthogonal to the vector towards your input point. This would be a question in its own right for Math SE but please search before you post.
