Trying to make a camera drag, but there's a weird initial jump, why?

I am using Rewired if that’s relevant, and I have a script that “knows” whether the right mouse button is held constantly (RewiredInputReader.RightMouseHeld += HoldCursor;) and I’m using a UIPointer script that, when the mouse is moved at all, sends out an event with the mouse’s coordinates, which it gets from the InputActionEventData’s GetAxis data.

The desired behaviour I am trying to get is what you see in the following GIF minus the jump at the start. I want the cameraFollower object to move against the mouse, as if you’re dragging a map:

any idea why I am getting this initial jump? Thanks

code

//I am using Rewired if that's relevant, and I have a script that "knows" whether the right mouse button is held constantly (RewiredInputReader.RightMouseHeld += HoldCursor;)
// and I'm using a UIPointer script that, when the mouse is moved at all, sends out an event with the mouse's coordinates, which it gets from the InputActionEventData's GetAxis data.

//I am receiving these two information in a script called ZoomFollowManager, like so:
private void HoldCursor(bool _heldOrNot) {
	if (followEnabled && !simulateGamepadMovement && !manualOverrideActive) {
		isDragging = _heldOrNot;
		initialMousePosition = new Vector2(newX, newY);
		initialObjectPosition = cameraFollower.transform.localPosition;
	}
}

private void DragCursor(float _newX, float _newY) {
	if (followEnabled && !manualOverrideActive && isDragging) {
		if (dragMultiplier > 0) { newX = _newX; newY = _newY; }
	}
}

//and I have a LateUpdate that makes the movements like so:
public void LateUpdate() {
	if (cameraFollower != null) {
		currentPosition = cameraFollower.transform.localPosition;

		if (isDragging) {
			Vector3 newMousePosition = new Vector2(newX, newY);
			Vector3 mouseMovement = newMousePosition - initialMousePosition;
			Vector3 move = initialObjectPosition - mouseMovement;
			move.z = cameraFollower.transform.localPosition.z;

			if (move.x < currentBoundaries.xBoundaryMinMax.x) {
				move.x = currentBoundaries.xBoundaryMinMax.x;

			} else if (move.x > currentBoundaries.xBoundaryMinMax.y) {
				move.x = currentBoundaries.xBoundaryMinMax.y;
			}

			if (move.y < currentBoundaries.yBoundaryMinMax.x) {
				move.y = currentBoundaries.yBoundaryMinMax.x;

			} else if (move.y > currentBoundaries.yBoundaryMinMax.y) {
				move.y = currentBoundaries.yBoundaryMinMax.y;
			}

			cameraFollower.transform.localPosition = move;
		}
	}
}

Turns out, the problem was the “isDragging” check inside DragCursor. removing that, it all works now as intended!