ScriptableBuildPipeline 1.21.20 build inconsistent result between using cache and not

To repro:
Create a simple project that contains addressable package

  • Create a spriteatlas that contains above two sprite
  • Mark the two sprite to “addressable”
  • Open Addressable groups window
  • Click [Build → Clear Build Cache → All] option
  • Click [New Build → Default Build Script] option
  • Click [Build → Update a Previous Build] option

The build results between 5 and 6 are different.

Looking into the modification of SBP 1.21.20, it seems that CalculateAssetDependencyData.cs results this issue.
Here is part of the code from that

        static internal ReturnCode RunInternal(TaskInput input, out TaskOutput output)
        {
            // ...

            HashSet<GUID> explicitAssets = new HashSet<GUID>(input.Assets);
            Dictionary<GUID, AssetOutput> implicitAssetsOutput = new Dictionary<GUID, AssetOutput>();
            HashSet<GUID> packedSprites = new HashSet<GUID>();

            Queue<int> assetsToProcess = new Queue<int>();
            for (int i = 0; i < input.Assets.Count; i++)
            {
                using (input.Logger.ScopedStep(LogLevel.Info, "Calculate Asset Dependencies"))
                {
                    if (cachedInfo != null && cachedInfo[i] != null)
                    {
                        AssetOutput assetResult = new AssetOutput();
                        assetResult.asset = input.Assets[i];

                        var objectTypes = cachedInfo[i].Data[4] as List<ObjectTypes>;
                        var assetInfos = cachedInfo[i].Data[0] as AssetLoadInfo;
                        var objectDependencyInfo = cachedInfo[i].Data[5] as List<ObjectDependencyInfo>;

                        bool useCachedData = true;
                        foreach (var objectType in objectTypes)
                        {
                            //Sprite association to SpriteAtlas might have changed since last time data was cached, this might
                            //imply that we have stale data in our cache, if so ensure we regenerate the data.
                            if (objectType.Types[0] == typeof(UnityEngine.Sprite))
                            {
                                var referencedObjectOld = assetInfos.referencedObjects.ToArray();
                                ObjectIdentifier[] referencedObjectsNew = null;
#if NONRECURSIVE_DEPENDENCY_DATA
                                referencedObjectsNew = GetPlayerDependenciesForAsset(input.Assets[i], assetInfos.includedObjects.ToArray(), input, assetResult, explicitAssets, implicitAssetsOutput, packedSprites);
#else
                                referencedObjectsNew = ContentBuildInterface.GetPlayerDependenciesForObjects(assetInfos.includedObjects.ToArray(), input.Target, input.TypeDB);
#endif

                                if (Enumerable.SequenceEqual(referencedObjectOld, referencedObjectsNew) == false)
                                {
                                    useCachedData = false;
                                }
                                break;
                            }
                        }
                        if (useCachedData)
                        {
                            assetResult.assetInfo = assetInfos;
                            assetResult.objectDependencyInfo = objectDependencyInfo;
                            assetResult.usageTags = cachedInfo[i].Data[1] as BuildUsageTagSet;
                            assetResult.spriteData = cachedInfo[i].Data[2] as SpriteImporterData;
                            assetResult.extendedData = cachedInfo[i].Data[3] as ExtendedAssetData;
                            assetResult.objectTypes = objectTypes;
                            output.AssetResults[i] = assetResult;
                            output.CachedAssetCount++;
                            input.Logger.AddEntrySafe(LogLevel.Info, $"{assetResult.asset} (cached)");
                            continue;
                        }
                    }

                    GUID asset = input.Assets[i];
                    string assetPath = AssetDatabase.GUIDToAssetPath(asset.ToString());

                    if (!input.ProgressTracker.UpdateInfoUnchecked(assetPath))
                        return ReturnCode.Canceled;

                    // Process uncached Sprites first, then all other uncached assets
                    var importer = AssetImporter.GetAtPath(assetPath) as TextureImporter;
                    if (importer != null && importer.textureType == TextureImporterType.Sprite)
                        output.AssetResults[i] = ProcessAsset(true, asset, assetPath, input, explicitAssets, implicitAssetsOutput, packedSprites, importer);
                    else
                        assetsToProcess.Enqueue(i);
                }
            }

            // ...
        }

As we can see,
while useCacheData is true, it simply set assetResult value from cachedInfo and call continue without maintain HashSet packedSprites; while useCacheData is false, it will pass packedSprites to ProcessAsset method, which will maintain and using it.