I do not totally understand your question, but for simple shapes a simple algorithm is ..
- just take the "center" of the object (the CG is fine) and make a vector from there to your vertex or triangle. That direction is "outside". If you're pointing over 90° away from that, you're pointing in to the guts of the object
the next more complicated approach ...
move along your normal say "one meter". call that point "test point". you should now be "outside" your building, correct?
now you just need to check if testPoint is inside or outside your 3D shape.
in short, to determine if you are inside or outside a 3D object, you just cast a ray and count how many times you intersect with the walls. if odd, you're inside, if even you're outside.
You can google hundreds of discussions on this on the www, example https://stackoverflow.com/a/63572837/294884
Note that there are many, many variations on this, and many problems too. In some cases you start from "just inside" your normal (ie go backwards a little); sometimes it's better to start from the middle of the object (if such a thing is knowable); sometimes there are issues about how the ray casting system works in edge cases (like "right on" the surface, which is what your vertex is)
the first solution seems to me to be the right one
It may be. Sometimes it is useless. It depends on the nature of your shapes.
I wanted to avoid having to do raycast ..
Can assure you - you will have to constantly raycast during any construction process! If you wish to "avoid" casting, set the idea aside. You will be casting until you are sick of it :)
I wanted to avoid having to do raycast to limit the generation time
The good news is this is totally incorrect. Casting is extremely trivial and a very minor burden compared to everything going on in a 3D scene. When you play any 3D scene, casts are being made 100s and 1000s of times each frame.
but if I have two buildings that are close to each other, the ray will find the wall but from the second building so I couldn't determine the normal is "inside" or "outside", right?
TBC in the simple system I described. You would certainly ONLY cast against that same building that you are working on! So that's the way to go.
But indeed, unrelated to what you're asking about. Say you're in a "city". You can indeed cast through "all walls" and the parity (ie: odd or even) will let you know if you are indoors or out. This has problems though, relating to edges, corners etc. But in some case it is relevant to that problem.
As mentioned on this answer https://stackoverflow.com/a/63593282/294884 ultimately you can investigate convex hulls and more. It's a big topic!
Note that often when you build dynamic buildings/shapes, you havce a data structure of each "wall" and indeed it knows which side is in and which side out. When you do that first, it's then easy to draw the triangles the correct way. It's a big topic!