Changing Scenes Causes Shadow Issues

I have two scenes in my project with different lights but (nearly) same lighting settings. If I die in one scene, it sends me back to the other scene. Going to this other scene by loading it from the scene I was in produces REALLY dark shadows. If I load the scene from the project files and then play it without loading up the first scene, the other scene’s lightings are fine.

After looking though the two lighting settings, the only difference is the Skybox lighting source. In one scene it is set to zero because I need it pitch black, and in my other scene (the one with the issue), it is set to one, because otherwise I get those really dark shadows.

How it’s supposed to look:

How it looks when I load in from another scene:

I guess what I’m asking is this:
Am I loading both scenes at the same time? I have the loading type set to Single, and it didn’t change.

The shadows are working right in both cases. Those holes you can see in the bottom picture are present in the top picture as well, probably caused by normal shadow bias, try to set that to zero on your light. The difference comes from ambient light.
Ambient light data is stored per scene. If you have multiple scenes loaded at the same time, the data from the active scene (with bold label) is used. Make sure that you have baked the ambient light for that scene. Optionally you could make sure that there are baked light probes and reflection probes present for the area.