changed to HDRP from URP and now raycast doesn't work when there are two cameras

I have two cameras rendering to two displays, and objects on different layers that are rendered separately on each camera.
I also have an object that is draggable(using raycasting), which was working just fine until recently when I decided to change to HDRP from URP.
Then the dragging stopped working. After looking everywhere and stripping my project down to a literal cube with the drag script and the two cameras I finally realized the problem was the second camera.
When the camera’s not active, the dragging works. When the camera’s active - it doesn’t even recognize the mouse is over the object - the raycast is not working.
Does anyone have any idea what’s going on?

same problem

I just switched, Yes it doesn’t work.