I want my canvas to react to my VR controller far ray interactor without blocking the raycast

What I hope to achieve is something similar to having the canvas group option to block raycast set to false, but still be able to interact with the UI.

The reason I want this is because the ray interactor changes lengths, which shifts its position when interacting with a UI element. I have a canvas filled with small buttons with small spaces between them. This results in having the ray jump all over the place, since its constantly switching between a long range ray to a short range ray.

Having the UI not block the raycast while also detecting inputs will solve this issue, is that possible?