Make sure it has a collider on it as well… Check the Project Settings to be sure your gravity has a non-zero value. Make a specific Physics Layer to try to work on. Other than that…mystery from your explanation. Happen in a new project?
Gravity works regardless the collider or the physics layer. Those have effect on collisions only. As @lightbug14 pointed out, the only setting I can think of that could cause that effect is the rigidbody constraint enabled in the Y position.