Pan camera in 3D scene

Question

Im writing a 3D application where I need to pan the camera using the mouse cursor. As in many 3D applications u press the mousewheel and move the mouse to pan the camera.

What Im doing is calculating the change in mouse location (dx, dy) to determine how much the camera should pan. To decide in which direction the camera should move Im casting a ray from the camera's point of view through the mouse position to get a directional vector. Next I calculate the perpendicular vector of the mouse ray using the cross product to determine on which axis the camera should move and move it accordingly based on dx & dy.

The problem:

When I look into direction 0, 0, -1 (into the screen) I can move the camera correctly along the X- & Y-axis.

When I look into direction 0, -1, 0 (into the "ground") I can move the camera correctly along the X-axis but it gets jittery around the Y-axis. Jittery as in it moves correctly to a certain point and then slows down or even goes into the opposite direction.

Another question is how would I go around moving the camera on the Z-axis since I only have the 2D mouse position dx and dy?

Below is the code Im using. Maybe my logic is wrong or Im overseeing something. Im not too familiar with vectors and matrices and learn while fiddling around with it so bare with me!

    //Calculate the change in mouse position
    float dx = (e.getX() - mouseX) * 0.02f;
    float dy = (e.getY() - mouseY) * 0.02f;

    //project the 2D mouse position to the 3D world position (mouse picking, raycasting) and normalise the vector to get a directional vector
    Vector3f direction = Maths.getRay(e.getX(), e.getY(), this).normalise();

    //Calculate the vector perpendicular to the mouse ray and the "up" vector
    Vector3f perpendicular = Vector3f.cross(direction, new Vector3f(0, 1, 0).normalise(); 

    //Move the camera along the axis
    Vector3f cameraPos = camera.getPosition();
    camera.setPosition(new Vector3f(cameraPos.x + (perpendicular.x * dx) , cameraPos.y + (perpendicular.z * dy), 0));

     //Cache the mouse position for the next iteration
     mouseX = e.getX();
     mouseY = e.getY();