Terrain Collision Only (No visible mesh)

Our game changes terrain altitudes frequently. We have a very customized terrain renderer that does not use a mesh, but uses a height map. It is important to be able to set terrain altitudes very fast.

I need terrain collision from an altitude height map (Texture2D) to let things bounce off of.

Is it possible to get at the underlying PhysX height map based terrain mesh or to make a Unity TerrainCollider without a Unity terrain object?

You can’t create a terrain collider without a terrain, but you can put the terrain on a specific layer and change the camera culling so it doesn’t render that layer.