IgnoreLayerCollision but then allow collision for some objects with that layer

I’m not sure why you’re experiencing a problem, but an alternative approach could be to use contact modification to selectively ingore/ not ignore collisions, like these balls that ignore the collision going in, but pay attention on the way out: