Removing a collider only for the player

Is there any way to make it so that only the player can pass through a collider but let anything else still collide with it?

I tried disabling the collider with

RaycastHit.collider.enabled = false

But of course this disables it for everything. Any way to disable it just for the player?