Need some help with Box colliders

I am not very sure if it is the problem with unity or box colliders, I setup some random cubes with box mesh collider and I control a player (a small ball in this case). My problem is if I play around it goes about normally But when I keep on movimg against the box collider it seems to going inside the box (kind of keeps wobbling there) and if I release it, It comes back to the outside edge of box collider. What should I do?

PS: I did not give the cube any rigid body physics But gave the player a rigid body component and set the collision detection to continuous as well. I tried a lot of other combinations to no avail.

Thanx in advance for your help.

reduce the box collider size .to make player go near to cube .u need to set the collider as same as the shape of a cube

Do you have the “Trigger” checked in the box collider component ?

Yes the box collider is exactly the same size as cube and the problem is that the sphere is going inside and bouncing up and down there, No I did not set the ‘is trigger’ option selected because if I select it, It is going right through it