Changing object scale without changing collider scale

Hi

Fairly new to unity but getting very into it and progressing well. I have a Unity 3d project with physics.

Example. I have a line of cubes, which I want to highlight when dragged over. This I have working in basic form. Using raycast hit and material change.

However, I would like to also shrink the object too to aid highlight selection (maybe even using animation). This though also shrinks the collider. Which in turns means object drops due to physics and causes holes in path of objects that I wish to raycast on. This then means missed selections or selection of objects in the distance.

Is it possible to resize the object without effecting the collider? I have tried both ways of object hierarchy but doesn’t seem to help.

e.g.

Mesh Renderer (master)
→ Box Collider / Rigid Body (child)

Box Collider /Rigid Body (master)
→ Mesh renderer (child)

I’m fairly new to unity as well, but what I’d imagine would work is to edit the Mesh Collider. Feel free to correct me if I’m wrong.