As an answer to my own question. (Question is still relevant )
Since I have started to suspect SSAO, I duplicated the “URP-performant” assets and enabled shadows for them.
I set this new settings as default for android and copied some more settings from default URP-performant.