Hybrid Graphics and Physics Raycaster

UI and objects coexist in a way where if I click drag a collider or a UI thing, it should drag the object.
As you can see, it doesn’t quite work.

It seems that canvas group doesn’t affect physics raycaster (? documentation) so the physics raycaster intercepts raycast events.

It seems that graphics colliders take over 3d physics objects even if those are in front.