Directional Light extreme shadows flickering/blinking buildup

I have an issue where after running the game for a while some shadows from the directional light of the scene starts to flicker lightly. It then slowly builds up over the course of about 5 minutes until every single shadow from any directional light source flickers uncontrollably. Here in the debug view you can see the light shaking uncontrollably. This shaking builds up over time.

It is HDRP version 10.6

It seems to not be related to the directional light itself, as I have tried deleting it and replacing it with a fresh one. It also seems to have nothing to do with Post-processing, since it continues after I turn it all off. The only way to make it disappear is to turn off the shadows of the directional light, or set its shadow dimmer to 0, but then of course I got no shadows at all.

As I already mentioned, it is as if something slowly builds up under the hood in some way.

I know it is not much to work with, but I would greatly appreciate any ideas of what might be going on.

how far is camera from origin (0,0,0)?

About (4000, 0, 4000)

I made some progress however. Turning of the enemies of the game makes the issue disappear. We are using DOTS for the enemies and are looking into it right now.

The camera rotation or something seems to affect the problem as well:
Project_Kryst - TPS_Scene - PC, Mac & Linux Standalone - Unity 2020.3.19f1 (gyazo.com)

Problem solved it seems.

It was indeed probably a floating point precision issue, however not due to the position of the light source in relation to the world. The problem was that we had enemies who fell through the ground and continued to fall infinitely over time. This caused the directional light to cast shadows on entities which where continuously further and further away.

So for anyone who encountered a similar issue and found this thread: make sure you do not have any objects in the scene which casts shadows very far from (0, 0, 0).