I've been working on rigid body simulation for a while and I still have no idea how to deal with this problem. (I use discrete collision detection and use LCP to solve the impulse of each contact point.)
Imagine a fixed bowl (a hemisphere centered at C) and a particle at the edge of the bowl. The particle rolls down on the surface to the inside of the bowl. And at some time step, there should be a contact point A, and the LCP solver will give the result impulse which makes the particle's velocity perpendicular to vector CA. But after updating one time step, the particle moves along this velocity a little bit and actually goes out of the bowl, and things get worse after several more time steps. When I used a cube instead of the particle, the cube can penetrate and sink through the bowl.
So is there a way to avoid this? The impulse method is not perfect and there can still be penetration after the collision response. I need to somehow correct the penetration, but simply moving the object along the surface normal at the contact point is not a good idea, because it can produce new penetrations.
Edit:
It is not the problem of bowl being too thin or time step being too large. The reason is the Euler's integral produce a polygon instead of a circle. And we cannot just modify the positions of these 2 object, because if there is a third object on the other side, object 2 and 3 may have a new penetration. I think one way is to add some elastic force based on the penetration depth, but it is not neat enough.
Thanks it's a good idea to look through the Bullet source code, working on it.