Problem with physics item pick up

Hey! Wanted to quickly come ask what’s the problem with my code here.

Basically what’s going on is that when I click left mouse button the ray does it’s thing, but for some reason the target object (the one this code is attached to and the one hit by the ray) just flings into space through walls and other stuff instead of being bound to the game object it’s supposed to get parented to. I can provide further info if asked.

Does anyone have a solution?

The function it’s supposed to do is to pick up the object hit by the ray and bind it to the ItemPickUp transform so that you can carry it around and drop it

public Transform ItemPickUp;
public float distance = 5.0f;
public Camera Camera;

void Start()
{
    Cursor.lockState = CursorLockMode.Locked;
    Cursor.visible = false;
}

void OnMouseDown()
{
    GetComponent<Rigidbody>().velocity = Vector3.zero;
    transform.parent = ItemPickUp.transform;
    RaycastHit hit;
    if (Physics.Raycast(Camera.transform.position, Camera.transform.forward, out hit, distance))
    {
        Debug.Log(hit.collider.name);
        GetComponent<Rigidbody>().useGravity = false;
    }
}

void OnMouseUp()
{
    transform.parent = null;
    GetComponent<Rigidbody>().useGravity = true;
}

Thanks!