How to prevent mesh to not go through other meshes

I use a Raycast to render a mesh where ever I look. However, when it renders any mesh I use, it goes through the other objects.

Code:

RaycastHit Hit;
        if (Physics.Raycast(direction.position, direction.forward, out Hit) && destinationPoint != null)
        {
            if (destinationPoint.transform.position != Hit.point)
                destinationPoint.transform.position = Hit.point;
        }

If I add Mesh Collider, it starts to move and rotate strangely and I’m also as a player moving too. How I can make this work? Doe anyone have a good proper solution to be able to render a mesh without going through other meshes but it can collides?

public float speed;
void Update(){

	Vector3 target = hit.point; // where object should be
	Vector3 source = trans.position; // where object is right NOW

	// object should travel vel(distance) in speed(time) to reach the target
	Vector3 vel = (hit.point - trans.position) * Time.deltaTime * speed; 

	rigidbodyOfObject.velocity = vel;
}  

tweek the speed value to get the good result