Character controller being pushed up by kinematic object

I’m working on a object interaction project where a character can pick up and drop certain objects. My FPS character is using a Character Controller for movement and the objects are using a Box Collider and RigidBody (set to kinematic when picked up) to help with the interactions. When my character holds and object and looks down the collision makes my character start to fly. Any information on canceling this type of collision or a better way to procceed? Thanks!

Disable the box collider when it’s picked up, unless it need to interact with other objects, then you can put it on a different layer or use IgnoreCollision