there are more ways of dealing with this problem but the safest I know of is to
Change ray direction slightly
this is numerically safe but implementation is not as easy as it sounds. Either change entire ray and compute from the start or change somewhere before the hit in question.

You need to ensure that you do not form close loops for example by zig zag pattern (so you alternate between turning CW and CCW within some cone from the original direction).
In case where ray is exactly parallel to and also touching an edge of your polygon either ignore such edge or count it twice or change the ray direction again.
Changing ray direction is always safe as it avoids singularities and numerical instabilities.
btw. This inside polygon algorithm you are using is well known by name Hit test