When I grab an object, it is too close

Hi, I have a problem and hope someone can guide / help me please.
Currently I can grab an object and the raybean works perfectly.
However, when I grab the object, it gets way too close.! Is there anyway I can set the distance the object is from my hand when I grab it? Please!

Captain_Pineapple, thanks for you reply. My obj is placed on top of pillar vertically. When I point the obj, it will come to my hand. For that I added Rigidbody, Sphere Collider and XR Grab Interactable components.
I have tried to offset the pivot point in Blender and reimport that new obj thinking it might work, it doesn’t. Thank you!

I am currently using Unity Version 2020.3.17f1