Culled light object still emits light

Using HDRP.

So I have a main camera, where everything gets rendered except a “handheld” object layer, those are rendered in a separate camera just for those so they dont appear to be clipping through walls (im using culling mask for this)

My problem is, if a light is on one side of a wall, and I go on the other side with a “handheld” object, the light from other side still affects it, even though that light is culled from the camera rendering the object, the ground on that side is dark because its getting rendered by the main camera which also renders the light.

I’ve looked up everything I can think of around this with no luck