If you have 8 perfect tracked points from two images and avoided the degenerate cases why is the essential matrix still wrong some times (or not good enough). Is this a numerical problem?
For any hint I grateful.
EDIT
This is my virtual scene. The marked points are used to calculate E and to check E as well.
Both views of the camera are merged in the image on the right. rel rotation around Y= -15° and rel translationVec set to [64, 114, 0]