Bounciness of 1 sticking to walls

Hey guys I have a problem with my current build of my Brick Breaker game. I am trying to make the ball bounce off of everything so I put a physics material on it which has a bounciness of one and no friction at all. None of the other objects have friction either. It works well except for one flaw. When is bounces it can greatly change velocity, which wouldnt be a problem except when the velocity is change to a very low number. When this happens and it tries to bounce against a wall it will just set the velocity to 0 instead of reversing it. This means instead of bouncing, I have a ball that just moves up and down a wall, or side to side on a top wall or paddle. I am not sure how to use these physics materials to my desired effect, and any help is appreciated. Thank you

The problem was the speed was below the bounce threshold. Moving that to 0 fixed the problem.