Any way to ignore collision between rigidbodies and colliders/character controllers?

i use the script below in my enemy bullet prefab. The thing is that because of the collider and the character controller i use on the player, the bullet does twice the damage it should. Is there a way to prevent collision with either the capsule collider or the character controller?

Make all character controllers exist on a different layer than the rigidbodies. Then, use the layer collision matrix to disable collisions between those two layers.