The resolution dialog stopped working after having built and ran it once with Hidden by Default. Putting it back on Enabled shows the dialog box but settings like Windowed are ignored. (build still starts in fullscreen and no scripting is telling it to).
Figured it out so might as well answer this myself if people come across it.
Go to player setting’s upper right menu and pick debug. A lot more options show up now. One of these is Default is Fullscreen and Default is native resolution, turn these off.
It seems like Unity doesn’t properly remove this toggle every time as it appeared as untoggled when not in debug mode. Or something in my project was preventing it to do so. Anyway, after doing this the resolution dialog had control over the game again.
I’m also getting the following error when in the player settings’ debug mode that could be related:
type is not a supported int value