Custom Spawning Netcode

Hello all, Im using Netcode For Gameobjects with the Facepunch transport.

I got my lobby system working great and starts my host or client based on server, then i swap scenes from menu to the game scene. The big problem Im having is if i just drag the player onto the network manager it spawns the player on the menu and again in in the loaded game scene. Then if i use a RPC like below it only spawns the player on the host and not the client (nether host or client players)

How can i fix this?

public class NetworkPlayerSpawner : NetworkBehaviour
{
    public GameObject playerPrefab;
 
    public override void OnNetworkSpawn()
    {
        SpawnPlayerServerRpc(OwnerClientId);
    }
 
    [ServerRpc(RequireOwnership = false)]
    public void SpawnPlayerServerRpc(ulong clientId)
    {
        //spawn 3D Prefab
        var tempObject = Instantiate(playerPrefab).gameObject;
        tempObject.GetComponent<NetworkObject>().SpawnAsPlayerObject(clientId);
    }
}