Unity navmesh, navigate to closest point of clicked object

See the image to understand what I am trying to explain.
I want to click on this tree and for the agent to find the closest point between it and the tree and navigate there, however it seems to just pick random points around each object and walks there.
The best thing I could find which could possibly help is NavMesh.FindClosestEdge however I can’t seem to get it to work with what I want to do with it. I tried on the Unity navigation forum and had no response in 48 hours.

if(Input.GetMouseButtonDown(0))
        {
            Ray ray = cam.ScreenPointToRay(Input.mousePosition);
            RaycastHit rayHit;
            NavMeshHit navHit;

            if(Physics.Raycast(ray, out rayHit))
            {
                float dist = Vector3.Distance(rayHit.point,transform.position);

                //For general ground navigation
                if(dist > .5f && rayHit.transform.gameObject.layer == LayerMask.NameToLayer("Ground"))
                {
                    agent.SetDestination(rayHit.point);
                }
                //If click tree!
                if(dist > .5f && rayHit.transform.gameObject.layer == LayerMask.NameToLayer("Tree"))
                {
                    if (NavMesh.FindClosestEdge(rayHit.transform.position, out navHit, NavMesh.AllAreas))
                    {
                        DrawCircle(navHit.position, navHit.distance, Color.red);
                        Debug.DrawRay(navHit.position, Vector3.up, Color.red);
                        agent.SetDestination(rayHit.transform.position);
                    }
                }
            }
        }

What i currently have above does not work, because i can only assume it’s looking for an actual navmesh hit rather than a gameobject hit. Any help would be so greatly appreciated, i really can’t understand how this isn’t a common issue with Navmesh games.

First of all, this line agent.SetDestination(rayHit.transform.position); sets the destination to the position of the gameobject that you clicked. This is why your agent is going to a random side, because your requested destination is right in the middle of the gap in the navmesh. Putting in navHit.position or even rayHit.point will likely yield more consistent results. If you want your player to go to the closest point to him, then an easy way could be to just nudge your destination towards your player, maybe something like

Vector3 playerDirection = transform.position - rayhit.transform.position;
agent.SetDestination(rayHit.transform.position + playerDirection.normalized);