Can't remove InputAction subscribed functions from destroyed objects

I have two scripts:
ControllerReference.cs that doesn’t get destroyed on load and has this reference to a button:
public InputActionReference ToggleMap;

and a MapController.cs that subscribes to ToggleMap at SceneLoaded:

  private void Awake()
        {
            SceneManager.sceneLoaded += OnSceneLoaded;
        }

        private void OnSceneLoaded(Scene scene, LoadSceneMode mode)
        {
            _controllerRef.ToggleMap.action.performed += ToggleMap;
          
        }

It also unsubscribes to ToggleMap in OnDestroy:

 private void OnDestroy()
        { 
            _controllerRef.ToggleMap.action.performed -= ToggleMap;
        }

There is one MapController in each scene, and inside ToggleMap() i only have a debug.log. When i go to scene 2 from scene 1 and press the ToggleMap button, i get 2 of the log instead of 1. and if i reference anything inside ToggleMap() such as debug.log(gameobject.name) it gives me nullref.

The problem seems to be that -=ToggleMap doesn’t do anything,and i get multiple += ToggleMaps that doesn’t go away even though the object with MapController.cs was destroyed in the scene change.

Please Help.

Somehow replacing _controllerRef which was set in OnEnable to be = ControllerReference.Instance, with just ControllerReference.Instance solved the problem. So by not using the stored variable in _controllerRef, but using what the _controllerRef would store directly?

Instead of _controllerRef.ToggleMap.action.performed += ToggleMap;
I now have ControllerReference.Instance.ToggleMap.action.performed += ToggleMap; and that works.