IPointer[Up/Down/Click]Handler Not Working When (Some) Sprites Are Behind It

I’m using 2D URP with the new input system, and trying to get GameObject clicks. Most work, but not all. Some seem to be blocked by sprites behind them.

I have an EventSystem.
I have a 2D ray caster on the Camera.
I have IPointer as the interface and the method implemented.
The objects are not children of a canvas, and there are no UI objects in front of them.
The layer order is correct (they are being drawn in front), and the sorting order is set.
These objects are prefab instantiated.
The objects have colliders on them (since some are working).

The up/down/click events work when the objects have scene instance (not prefab) objects behind them (and the objects are children). However when I have other prefab instantiated objects behind, they are not registering.

Does anyone have any thoughts why this may be?

Roughly:

Scene

  • UI
  • World
  • Prefab (Big): Sort: Default, Order: 1
    ± Prefab (Small to be clicked): Sort: Resources, Order: (random positive number)

Layers:

  • Background
  • Default
  • Resources

I tried doing it without Prefabs (instead instantiating clones from what is already in the scene), but I’m having the same result. Basically, with the following setup, I’m having the same issue:

Scene

  • UI
  • World
  • Template (Big): Layer: Default, Sort: 1
    ± Template (Small to be clicked): Sort: Resources, Order: (random positive number)
  • Clone (Big): Layer: Default, Sort: 1
    ± Clone (Small to be clicked): Sort: Resources, Order: (random positive number)

Sort Layers:

  • Background
  • Default
  • Resources

I managed to work around it by setting the layer (top next to tag) of the blockings objects to “Ignore Raycast”, and selecting only the “Default” layer on the camera raycaster.

Things should be working without this “tweak”, so something is wrong. I’d like to get to the bottom of it, any thoughts on how to debug further?