For some reason my player slowly goes up in the air like a baloon at about 0.05 per second…
If I put the mass any higher than 1 then it slowly goes down by 0.05 per second… So if the player walks down a hill he just glides… no matter what his mass, it could be 5 or 500 it still falls at an incredibly slow rate?

If you terrain if flat, you can do something like :
function Update ()
{
transform.position.y=groundLevel;
}
groundLevel is the level of the ground.
Hope that will help you.
You should try to lerp the object velocity to zero and see if that helps. I suggest putting this when the player doesn’t input any movement controls
void Update()
{
if (Input.GetAxis ("Horizontal") != 0.0f || Input.GetAxis ("Vertical") != 0.0f)
{
// This is just an example of the player is inputting any movement
}
else
{
// The player is not inputting any movement
rigidbody.velocity = new Vector3.Lerp(rigidbody.velocity, Vector3.zero, Time.deltaTime);
}
}
Also, follow what jmgek suggested, try to disable gravity and use scripted movements. For gravity-like movement in the air you could use rigidbody.AddForce downwards (or lerping velocity)