How to keep objects from passing through colliders

I have set up a simple holding system so when the player looks at an item and holds E it lerps the object 2 units in front of the camera. Now my problem is that when I am to close to a wall the object passes through it because it is staying 2 units in front of the camera but is glitching through the colliders in the process. What I’m trying to achieve is that item will listen to the code but also continue to collide similar to Half-Life, Portal, Elder-Scrolls holding system. I hope this all made sense, I’ll attach an image of the problem, and thanks for reading

After some experimentation I found that using smooth damp instead of lerp keeps it from going through the wall. But then the cube still was trying and would shake violently so instead of using transform.position I used rigidbody.position which removed most of the shake. Thank you and I hope this helps somebody else.

Usually when objects pass trough colliders, you may have used a transform property. Like rotation, or translation. Or probably the colliders do not contain the objects inside themselves. I mean they are outside the colliders.
I am not sure what is Lerps, and I havent played Half Life, or any other of those games.
Maybe you can put a collider to the objects themselves that are trespassing.
If you explain a little more maybe i can help you.