Please use code tags .
I notice you are using transform.forward and transform.right as your movement directions, but nullifying the Y movement. My best guess is that rotating the camera has changed the projection of transform.forward onto the XZ plane, so that most of the “attempted” movement is now along the Y axis, and only a little along X/Z.
One way to deal with that would be to normalize the projection before you multiply by deltaTime and panSpeed. (Note: that means you’d need to zero the Y first, rather than at the end.)
Another option is to change the way that you are calculating the direction of movement so that it doesn’t depend directly on transform.forward and transform.right. Maybe start with up and right relative to the world (rather than the camera) and then rotate them by one of the camera’s euler angles or something.
Also, I don’t know what you’re trying to do with the moveDirection variable. In the first part of your code, it seems to just be a way of saving the camera’s previous Y coordinate, and in the later part you do some math on it but then never appear to use the result for anything.