In the documentation, info on Event Raycasters is rather sparse - I can’t figure out if PhysicsRaycaster is meant to substitute the pointer as the focus of input from Graphic Raycaster or I need to actually do something with the input module. I want to use the center of the camera to control World Space GUI in a simple FPS - panels and buttons on every door, floor, roof; everything in beautiful interactive glory! 
I think you need a custom Input Module, rather than a raycaster - you want to feed in a custom pointer position (center of the screen).
Hey,
You could center pointer and lock it to the screen. Hence avoid all the headaches.
At least in theory it sounds doable