NavMeshAgent calculated path going in a straight line ignoring height in between start and end

I’m trying to use the NavMesh system to calculate the path only and then move my objects manually along that path.
I’m doing this by disabling the automatic updates of position, rotation, upaxis and setting obstacle avoidance to none, then setting agent.nextPosition to the start position and calculating the path using agent.calculatePath with the destination.
Then I get my array of points that make up the path like I want.
The problem is, it completely ignores height and goes straight through humps or over valleys in a straight line. But I can walk on top of a mountain if I set the destination to be there, it just goes in a straight line up instead of curved.
Why? What am I missing?

Edit: Here is a video to illustrate the problem (ignore the colorful balls :slight_smile: )

Did you set a max step height and slope angle before baking the navmesh?

I left it at the defaults, 0.5 step height, 45° angle.
Also tried modifying it but no difference. (Re-baking after every change).
Why is there a seperate “Baked Agent Size” on the Bake Tab after already defining a named one in the Agents Tab? I set both to the same values.