Managing addressable and not-addressable assets

Unity Engine
1

Hi there.
So finally I decided to give Addressables a try and the first thing which I wanted to switch to Addressables (from direct references) is the fonts. In our game we have a localization system which apart from localized text stores the fonts used for different languages (consider it as a simple dictionary/map “language”-“font”). The rationale is that if, for example, the player uses English he/she does not need a Chinese font loaded into the memory.
So I came up to the following code:

public void GetFontAsync(SystemLanguage language, Func<Font, Font> callback)
{
    var fontAsset = _fontAssetContainer[language];
    var operation = fontAsset.LoadAssetAsync();
    operation.Completed += (selfOperation) => OnFontAssetLoaded(selfOperation, callback);
}

private void OnFontAssetLoaded(AsyncOperationHandle<Font> operation, Func<Font, Font> callback)
{
    if (operation.Status == AsyncOperationStatus.Succeeded)
    {
        var oldFont = callback.Invoke(operation.Result); // pass the new font, get the old one
        if (oldFont != null)
            Addressables.Release(oldFont);
    }
}

The problem is that oldFont could be a font which was loaded through the direct reference in scene and therefore I get several warnings “Addressables.Release was called on an object that Addressables was not previously aware of. Thus nothing is being released”. I believe that this warning is harmless by itself but it indicates a potential problem: if Addressables.Release is called with directly_referenced_font after this font is loaded through addressables the release will actually happen and that’s probably not what I actually want.
So how to deal with such situation?

2

Any ideas?