Use cv2 aruco marker pose estimation to get camera position in vtk

Question

I am trying to find a camera position based on the

cv2.aruco.estimatePoseSingleMarkers 

function. I have placed a marker on a real object of which i also have an stl. The Marker is placed on a position such that if i load the stl into vtk the markers coordinate system aligns with the world coordinate system. No my intention is to get the rvec and tvec using a real world camera and use them to place the vtkcamera s.t. I have the same scene in vtk as I have in real life.

Right now I am trying it by

rvec,tvec=calc()
tvec = -1*np.array(tvec)
rmat, _ = cv2.Rodrigues(np.array(rvec))
rmat=rmat.transpose()
rotated_tvec = np.dot(rmat, tvec[0].transpose())

The rotated_tvec gives me a pretty good estimate for the position. However I am not sure how to rotate the camera correctly. I have tried something like this

transform2 = vtk.vtkTransform()
camera.ParallelProjectionOff()

rotation = vtk.vtkMatrix4x4()
for i in range(3):
    for j in range(3):
         rotation.SetElement(i, j, rmat[i, j])
rotation.SetElement(0, 3, translation[0][0])
rotation.SetElement(1, 3, translation[0][1])
rotation.SetElement(2, 3, translation[0][2])
rotation.SetElement(3, 3, 1)
    
transform2.SetMatrix(rotation)

camera.ApplyTransform(transform2)

Where I used both tvec and rotated_tvec as translation, however that does not work. Any help would be appreciated.