Baked lightmap shadows jitter on screen edge in quest 2 standalone

I have a scene with baked lightmaps. This scene works fine in the unity editor, and when testing using the link cable, the shadows work fine.

The issue is in standalone mode, the shadows in the distance for certain objects become jittery and seem to z fight objects that are distant from them.
This is a video that show the issue in action:
Link: Shadow glitch issue - YouTube

I managed to avoid the issue by making my polygons smaller.

As in before I had a single plane stretched out to cover the whole length, of the walls / ceiling. I added loop cuts to it and the issue was gone.