Rotating a 3D object on the Y axis relative to the cursor's position?

I’m trying to combine the implied camera perspective of classic 2D Zelda titles with the controls of twin-stick shooters, in 3D. I can move the player with the WASD keys, but I’m not sure how to rotate him to face the mouse cursor. I think I need to do the following:

  1. Calculate the cursor’s position on the screen.
  2. Draw a line between that position and the player character.
  3. Calculate the difference between the player character’s current Y axis rotation and that line.
  4. Rotate the player character on the Y axis by that difference, so that the difference becomes 0.

This way if the player moves the cursor to the top of the screen the character will look “North”; if the player puts the cursor in the bottom left corner, the character will look “Southwest”, if that makes sense.

I tried using transform.LookAt, but this caused a crazy feedback loop that made the character spin out of control:

  Ray mouseRay = playerCamera.ScreenPointToRay(Input.mousePosition);
  float midPoint = (transform.position - playerCamera.transform.position).magnitude * 0.5f;
  transform.LookAt(mouseRay.origin + mouseRay.direction * midPoint);

I also tried the Brackey’s tutorial here, but was unable to calculate the lookDir because ScreenToWorldPoint returns a Vector2, while a 3D object’s transform.position is a Vector3.

Any ideas?

I’d raycast against a mathematical plane at the height of the player, then use look rotation!

Ray mouseRay = playerCamera.ScreenPointToRay(Input.mousePosition);
Plane p = new Plane( Vector3.up, player.position );
if( p.Raycast( mouseRay, out float hitDist) ){
	Vector3 hitPoint = mouseRay.GetPoint( hitDist );
	player.LookAt( hitPoint );

So sounds like you just need help with the player looking at the mouse.
This should work out:

    void Update()
        Ray ray = cam.ScreenPointToRay(Input.mousePosition);
        RaycastHit hit;
        if (Physics.Raycast(ray, out hit))
            transform.LookAt(hit.point); // Look at the point
            transform.rotation = Quaternion.Euler(new Vector3(0, transform.rotation.eulerAngles.y, 0)); // Clamp the x and z rotation

It will the player rotate on the y-axis only. To be honest, I’d add a Vector3.Lerp just to make it more smooth! ^^