Object not passing through a collider properly

I’m doing a game where the player switches between layers and the level changes depending on the current layer. The game hides everything that isn’t in the same layer as the player.

Here is the problem: Because the layers have objects that need to collide with each other I can’t disable the colliders when the player is in another layer.

I’ve tried using both kinds of collision ignore but neither work. When the player isn’t in a matching layer collisions still happen. Although if the player is standing on a ignored collider, the collider acts like quicksand. The player will go through if you walk back and forth on the collider.

The player has a Rigidbody and a CapsuleCollider and the ignored objects have BoxColliders either in parent or a child. and some of them have rigidbodies also. CollisionIgnore doesn’t work with any kind of object.

I finally figured out what caused the problems. The ground check of the character controller has a different layer than the player so it collides with unwanted layers.