Collider thats used only for Raycasting

Right now the characters in my game have small colliders attached to them in their character controller for their movements, however I would like to add a much larger collider that only receives Raycasts to help the user select characters easier.

Is there a way to have a collider attached to an object which doesn't affect the physics of an object, but still receives RaycastHit's?

Click the Is Trigger checkbox on the collider, it should stop it affecting physics. If you want to stop it from firing trigger events, you can set your object to use a layer which you then disallow collisions with for all other layers (in Edit > Project Settings > Physics)