Strange Interact Destination

Hi guys, I have problem that im trying to work out for few hours without solution. Here is a gif and explaination of the problem:

The problem is: im right clicking (interaction-move) each of those 3 blocks. The thing is character is always running to one specific point to interact, no matter what. I would love to make it that character is running to closest point of the object, not that strange point. Its super annoying If I have big wall and character run around it to the middle on the other side…

My related code:

public NavMeshAgent playerAgent;
public void MoveToInteraction(NavMeshAgent playerAgent)
    {
        this.playerAgent = playerAgent;
        playerAgent.stoppingDistance = 3f;
        playerAgent.destination = this.transform.position;
    }

    NavMeshAgent playerAgent;
    public GameObject objectToSpawn;

    void Start()
    {
        playerAgent = GetComponent<NavMeshAgent>();
    }

    void Update()
    {
        if (Input.GetMouseButtonDown (1) && !UnityEngine.EventSystems.EventSystem.current.IsPointerOverGameObject())
            GetInteraction ();
        bool isRunning = false;
        if (playerAgent.remainingDistance - playerAgent.stoppingDistance >= 0.1)
            isRunning = true;
        else
            isRunning = false;
        GetComponent<Animator>().SetBool("IsRunning", isRunning);
    }

    void GetInteraction()
    {
        Ray interactionRay = Camera.main.ScreenPointToRay (Input.mousePosition);
        RaycastHit interactionInfo;
        if (Physics.Raycast(interactionRay, out interactionInfo, Mathf.Infinity))
        {
            GameObject interactedObject = interactionInfo.collider.gameObject;
            if (interactedObject.tag == "Interactable Object")
            {
                interactedObject.GetComponent<Interactable>().MoveToInteraction(playerAgent);
            }
            else if (interactedObject.tag == "Ground")
            {
                playerAgent.stoppingDistance = 0;
                playerAgent.destination = interactionInfo.point;
                Instantiate(objectToSpawn, interactionInfo.point + new Vector3(0, 0.1f), Quaternion.Euler(90, 0, 0)); // click animation on the ground
            }
            else
            {
                playerAgent.stoppingDistance = 0;
                playerAgent.destination = interactionInfo.point;
            }
        }
    }

Rescue me pls. Help in advance!

Is your MoveToInteraction pathing to the closest location on the NavMesh from the center of the object?

If so; internally it’s finding the same point each time because it will always be the closest point to the center.

It kinda sounds like you have this issue.

Yea I think it’s the same issue. I can’t even make doors as interactable objects because they need to have collider’s and my character is trying to open them always from the same side.