Same prefab has different Raycast collision interaction based on position (2D)

Hi, I am currently fixing some physics related issues with our raycast movement and noticed an interesting thing. Our levels have different interaction with the physics depending on if it's the first level (situated at 0,0) or something other than that. Our map generation system uses level prefabs to generate levels and I made it so that it only has acess to one level for testing purposes, namely:
9846621--1417203--upload_2024-5-21_12-2-38.png

Now for the visual of the problem itself. In the first level it works very good with no issues with collision:
9846621--1417206--Example 1.png
It never comes close to "entering" the collision. Now here is that same exact collision but on another level:
9846621--1417209--Example2.png
It enters the collision itself (causing normals and bouncing to break and possibly trapping it inside)
Here the two levels used for testing can be seen:
9846621--1417212--Full map.png

Both levels use tilemap colliders for the rocks
What could be causing this and how can it be solved?

So it seems that the position of the levels was the trigger for a certain issue to occur, they did not have anything to do with eachother really.
I used RayCast2D (not the multiple return one) and I did not account for the fact that the raycast would not trigger again if it hit the player (even if I accounted for it), I gave the raycast a mask and removed the player collision from the chance if triggering it alltogheter and now it seems to work. What I assume is that the position of the rooms somehow made it trigger more when it was further away from 0,0.