How do I make a rigid body object ignores certain colliders?

I have a rigid body object which can go through walls, but cannot through floor (such as a ghost). How do I implement this?

Use Physics.IgnoreCollision, where you ignore the walls but not the floor.