It is fast because the usage of the Lerp method is wrong. For the third parameter your argument is
Time.deltaTime * zoomSpeed
The way the lerp works is, if this parameter is 0 it returns your first argument camera.orthographicSize. If it’s 1 it return your second argument. The values in-between interpolate between your both arguments. Hope it makes sense. So what you want to do is, map the zoom value between 0 and 1, modify that value with the scroll wheel and then use that as an argument for the lerp. Let me know if it doesn’t make any sense and I can provide a code example in my free time.