The following assumes you already know that the center of the circle is inside the polygon. There are a few things you want to check as your definition that touching the vertices is ok adds some cornercases. This solution works for concave polygons as well.
Early check
For the circle to be fully inside the polygon, we need all edges to be outside the circle. In particular this ensures that the polygon is not fully inside the circle.
Given the circle of radius r and centered at c and the edges e0, e1, ..., en of the polygon, a necessary condition is thus that for all i < n:
d(c, ei) >= r
where d is th euclidian distance.
If the above does not hold for any edge, then either there is an intersection between the polygon and the circle or the polygon is itself fully inside the circle.
Does the circle intersects the polygon
The last check is a necessary condition for the circle to be inside, although it is not sufficient as it is possible that all edges be outside the circle, but that the circle still leaks out of a vertice.
Let's first remember some formulas we will need.
Equation for a circle of radius r centred at (x0, y0):
(x - x0)2 + (y - y0)2 = r2
Thus the intersection with a line y = ax + b is found by solving:
(x - x0)2 + (ax + b - y0)2 = r2
This is nothing but a quadratic equation that can be rewritten as:
(a2 + 1)x2 + (2ab - 2ay0 - 2x0)x + (x0 + (b - y0)2 - r2) = 0
You can solve that with the quadratic formula for each vertex. You then have three possibilities.
1) There is no solution
This indicates there exists no intersection with this vertex. With high-level languages, you can catch some kind of MathError
exception to detect that. Otherwise, you can mathematically check the sign of the discriminant as this case happens if it is negative.
(2ab - 2ay0 - 2x0)2 - 4 (a2 + 1) (x0 + (b - y0)2 - r2) < 0
2) There is a unique solution
If the equation has a single solution, that is both solution are the same, then the circle may touch, but does not leak out of the edge. You stated this is still considered to be inside the polygon in your case.
Mathematically, this happens when the discriminant is zero.
(2ab - 2ay0 - 2x0)2 - 4 (a2 + 1) (x0 + (b - y0)2 - r2) = 0
3) There exist two solutions
If there exist two solutions, say xi and xj, then there might be an overlap. Although, this is not certain in the case of concave polygons.
To check if there actually was an overlap, you must check if the intersection happens on your line segment.
This is quite simply done. Suppose your vertex lies between the points (x1, y1) and (x2, y2), then there is an intersection if an only if...
x1 < xi < x2
or...
x1 < xj < x2
In any other case, the intersection happens on the continuation of the vertex, not on the vertex itself.
If one of the above condition holds true, then and only then do you know that your circle leaks outside the polygon.
Final cornercase: concave edges
As stated, touching the polygon is ok, and thus there is a final cornercase not covered by the above: touching a concave edge.
A concave edge is an edge which inner angle is bigger than 180°. Thus whenever there is an intersection with the polygon, you want to ignore it if the intersection happens on an edge that is concave.
All of the above works for any polygon, not only triangles and hexagons.