How to implement camera movement by dragging when using a perspective camera

Hey there! I have lately changed my camera from orthographic to perspective in my 2D game and since I did that dragging the camera around doesn’t work anymore.

I did some research and stumbled upon a few thread in this forum but none of it helped. Currently my code for the camera drag movement looks like this:

void DragMovement()
{
    if (Input.GetMouseButtonDown(2))
    {
        originPos = cam.ScreenToWorldPoint(new Vector3(Input.mousePosition.x, Input.mousePosition.y, cam.nearClipPlane));
    }

    if (Input.GetMouseButton(2))
    {
        Vector3 posDiff = originPos - cam.ScreenToWorldPoint(new Vector3(Input.mousePosition.x, Input.mousePosition.y, cam.nearClipPlane));
        cam.transform.position = ClampCamera(cam.transform.position + posDiff);
    }
}

So, basically, the camera should be moved around when holding the mouse wheel down but its not working.

Any idea on how to fix this?