Basic Vector calculations

In this top down game, I have the player changing his rotation based on where the mouse position is, however I’m having a hard time understanding this line of code right here.

targetRotation = Quaternion.LookRotation(mousePos - new Vector3(transform.position.x,0,transform.position.z));

So from what I understand if I subtract the mouse position from the player’s position I will get the position of the mouse, plus wherever the player moved to. However I’m having trouble understanding the actual math behind it. If my mouse position is at (0,.6, 1) and I subtract that by the players position on the x and z axis(for example (5,0,5), what is the new position that I get. I’ve tried to write it out on paper but the math never adds up for me.

void ControlMouse()
	//Player rotation
	Vector3 mousePos = Input.mousePosition; 
	mousePos = cam.ScreenToWorldPoint(new Vector3(mousePos.x,mousePos.y,cam.transform.position.y - transform.position.y));
	targetRotation = Quaternion.LookRotation(mousePos - new Vector3(transform.position.x,0,transform.position.z));
	transform.eulerAngles = Vector3.up * Mathf.MoveTowardsAngle(transform.eulerAngles.y,targetRotation.eulerAngles.y,rotationSpeed * Time.deltaTime);

	//Player Motion
	Vector3 input = new Vector3(Input.GetAxisRaw("Horizontal"),0,Input.GetAxisRaw("Vertical"));
	Vector3 motion = input;
	motion *= (Mathf.Abs(input.x) == 1 && Mathf.Abs(input.z)== 1)?.7f:1;
	motion *= (Input.GetButton ("Run")?runSpeed:walkSpeed);
	motion += Vector3.up * -8;
	controller.Move(motion * Time.deltaTime);

To get the AB vector, you need to compute B - A.
In this case,

mousePos - new Vector3(transform.position.x,0,transform.position.z)

gives you the direction from the player transform to the mouse position, which is exactly what you want for Quaternion.LookRotation, because it needs the forward direction of the rotation you want. That means Quaternion.LookRotation will make the player look in this direction, which is towards the mouse.
Does that somehow help you ?