My ingame avatar's worldspace does not coincide with other objects

I have a 3D world with an avatar. I am tracking the avatar’s right hand and once it is hovering a Canvas UI element, I want to interact with that object. My issue is currently to translate these positions properly.

I’m using a GUI script to follow the world position of my mouse, and I am currently also logging the world position of my tiles. Hovering over the tiles with my mouse, gives the same world position.

However, these positions do not correlate with my avatars hand position. I tried to translate my rightHandPosition using ScreenToWorldPoint, but the values are still wrong.

var anchorPos = RectTransformUtility.WorldToScreenPoint(cam, rightHandPos);
var preciseScreenPointToWorld3 = cam.ScreenToWorldPoint(new Vector3(anchorPos.x, anchorPos.y, cam.nearClipPlane));

This code gave me the proper arm world points.