How to animate camera in React three fiber without lookAt()?

Question

I am trying to animate a camera movement so that when a user clicks on the contact menu, the 3D model shows on the left of the screen. Something like this project

Currently, I'm using the UseFrame hook as in this tutorial

The problem is I'm using the lookAt() function and the 3D element is always in the center of the screen.

Here's my code

import { useAnimations, useGLTF } from '@react-three/drei'
import { useEffect, useRef } from 'react'
import { useControls, button} from 'leva'
import { useFrame } from '@react-three/fiber'
import { Vector3 } from 'three'

export default function Character ({isContact}) {

    const character = useGLTF('./model.gltf')
    const { actions } = useAnimations(character.animations, character.scene)
    
    const markerRef = useRef()
    const vec = new Vector3()
    
    useFrame((state, delta, xrFrame) => {
        
        if(isContact){                           
            state.camera.lookAt(markerRef.current.position.x , markerRef.current.position.y + 1  , markerRef.current.position.z  )            
            state.camera.position.lerp(vec.set(-4, 0, 0 ), 0.01)
            state.camera.updateProjectionMatrix()
        }
        else if(!isContact){            
            state.camera.lookAt(markerRef.current.position.x , markerRef.current.position.y + 1 , markerRef.current.position.z )
            state.camera.position.lerp(vec.set(-3, 1.5, 4 ), 0.01)
            state.camera.updateProjectionMatrix()
        }
        return null
    })
    
    
    return (
        
        <primitive 
            object= { character.scene}
            position-y= {-1.2}
            ref={markerRef}
            
        />
        
        
    )
    
}

I also tried to add +2 at the z coordinate in the lookAt() function like so:

state.camera.lookAt(markerRef.current.position.x , markerRef.current.position.y + 1  , markerRef.current.position.z +2 )                                    

The position is correct, but the problem is that the camera animation "bounces"

Any thoughts on how to do that?