Strange specular lighting in fully shadowed areas

I have a scene that is mostly static, it’s in a city at sunset with the sun quite low and warm. I’m trying to bake and get some nice gradients in the indirect diffuse lights. However, the specular lighting seems to come from nowhere and screw with ground to wall corners in a way that I don’t really understand.

Here is a screenshot from a similar angle to where the sun is coming from (you can see it selected), and you can clearly see the light hitting the upper parts of the building and get overexposed, just like I want.

But look at the ground near the wall, why is it brighter there? There should be no sun hitting in a way that would reflect on the ground like that.

Here the same scene from a different angle with SpecularLighting selected in the HDRP debug menu, so it is some form of specular light. Why is this showing up way off from where the sun is hitting? I understand that this is not raytraced, but the whole thing is in shadow? And It’s like in the opposite angle from how the sun is hitting.

Here is a picture of just the indirect diffuse light, looks good. The bright and non bright parts of the ground and wall have even switched place…

What am I missing here?

I think I figured it out. I had a reflection probe in that area that was placed to high. So the specular lighting was sampling from perspective of being 10m over ground, where the sun hit quite different than on the ground. I did not realize the specular light was so affected by a misplaced reflection probe.