When something is not serialized, the value of it stays zero (because the memory allocated for the class instance is zeroed before anything is done to it, and serialization overwrites only values that have been seralized). So, for a MonoBehaviour, if it is not public and it does not have [SerializeField], it will always be zero/null immediately after control of the instance is passed to your code, and if you enter/exit play mode, it is deserialized again, and again the memory is zeroed.
Now, the ScriptableObject is not being deserialized each time, because it is an asset rather than a scene object, so I actually would not have been surprised at this behaviour on its own, if the non-serialized fields kept their values in the editor, because Unity would just maintain one instance. HOWEVER, if you put [System.NonSerialized] on the field, this does not happen, and it gets reset every time.
I did a test: I made a bunch of ints (one public, one internal, one protected, and one private). I made a button that incremented them all, and I made a button that printed them all. I did a ScriptableObject version, and a MonoBehaviour version. Results:
MonoBehaviour - all changes to all values (public or otherwise) lost when exiting play mode (this is correct behaviour)
ScriptableObject with no attributes - all changes to all values (public or otherwise) maintained when exiting play mode but lost when exiting Unity (this is not what I want to happen but it is what I would expect to happen)
ScriptableObject with NonSerialized attributes on non-public fields - changes to public fields maintained when exiting play mode, changes to non-public fields lost when exiting play mode
I can think of no behaviour that supports this: If the asset were being serialized/deserialized every time, then even without the NonSerialized attribute, the values should not persist. If the asset were not being serialized/deserialized every time, then the NonSerialized attribute should have absolutely no effect because it’s an in-memory copy.
I’m starting to think this is some sort of dirty hack one of the developers put in deliberately for this situation, maybe somehow for using ScriptableObjects as editor-utility objects? Because there is no reason that I should be telling Unity “don’t do this thing you weren’t going to do anyway” and have it actually have an effect.