Switch to 3D world map like Skyrim

Hi everyone, just wanted to get some thoughts on what would be a good approach to having a 3D World map like you have in Skyrim. The main challenge is that the player is already in a level with its own camera and lighting setup and then can switch to and from the world map at any time.

My guess is that the 3D world map should be loaded as a persistent (don't destroy on load) object with its own camera and lighting setup that only targets a "World Map" layer. When the player opens the map, it should just turn off the current game camera and enable the World Map camera. The whole world map will then be loaded while the current game level is still active in the background.... if that makes sense...

Any thoughts and ideas would be appreciated!

I put the player and camera in it's own scene that's always loaded and then load/unload maps in and out.