Collider doesn`t work as it should, misbehaves randomly

Hello everybody,
I am working on a game, and a part of it consists of a bridge that can be trespassed only once. The bridge consists of multiple bricks, and after you exit a brick, it disappears behind you. However on the other side of the bridge, by pressing a button, you can rebuild the bridge again and walk it once more. I tried solving this via a box collider for the bricks. On trigger exit, the brick should disappear with gameObject.SetActive(false). When the button is pressed, it is set to active.
Now to my problem: this solution of mine works, however it misbehaves randomly. I try walking back and forth multiple times and after some attempts, some bricks dont react anymore. I also tried inputing a Debug.Log if the collider on each brick is entered, and in the aforementioned bricks, the Debug doesnt show. So the problem begins with detecting the collision, not the disappearing part. What worries me is that it happens completely random and without any noticable pattern.
Do you have any tip or idea as for what I should look for in the script or what could be the reason for such behavior?
Thanks in advance!

There is not much detail to go on here, but it is important to know that when you disable a GameObject any physics objects on it does not trigger Collider.OnTriggerExit. Maybe that can help you solve it?