How to only load certain clients to a scene UNITY MULTIPLAYER

I have a multiplayer game which uses a Server and i’ve come across a problem.

I cant load only certain Clients into a game/scene. I still want the server to have access to all the different games, but when i try to load a scene with the NetworkSceneManager all the clients are loaded into every single game/scene.

I would appreciate any help