ScriptableBuildPipeline - Known issues, roadmap, ...

[pavelkouril]

Hello,

I’ve been testing SBP to build asset bundles (jncluding integrating the new cache server), and I’ve been hitting issues when building asset bundles (which will get built correctly using the legacy built-in pipeline).

Therefore, I would like to ask if there is a list of known issues (e.g., some scenes/prefabs references to scripts will get broken, empty paths when using the cache server integration, etc.) with the current pipeline, which could save me hours/days of time by trying to hunt the issue in a big project.

Also do you have some approximate roadmap? Especially, when the package is expected to go out of preview?

Anything will be helpful, really.

[Ryanc_unity]

@pavelkouril The documentation included with SBP details what differences to expect (such as variants not being supported). Everything else should work, and if not, it’s a bug we need to fix (and no aware of) so please open a bug report for the issue. We did just push out a new version: 1.1.1-preview that fixes issues with asset bundle’s that contain scenes.

As for roadmaps, we are not ready to share anything just yet in this area, but going out of preview should be soon.

[pavelkouril]

Well,

for one we are hitting an issue I submitted as case 1096354; unfortunately without repro case, because I don’t have any idea how to reproduce this.

The other issue was getting errors when using cache server with the SBP and prefabs/GOs in scene referenced non-existing Mono Behavior (but the component was still there); we fixed this on our side by removing the references, and I will try to prepare a repro case once I’ll have an hour or two of free time and will submit it using the reporter.

[Ryanc_unity]

Are you using the same CacheServer for SBP and your Project?

The SBP docs should call out that you need to use a separate CacheServer for Build Artifacts from Project Artifacts otherwise a collision will occur as the CacheServer was originally not designed to handle two different sets of artifacts for the same asset.

[pavelkouril]

@Ryanc_unity

Are you using the same CacheServer for SBP and your Project?

The SBP docs should call out that you need to use a separate CacheServer for Build Artifacts from Project Artifacts otherwise a collision will occur as the CacheServer was originally not designed to handle two different sets of artifacts for the same asset.

I was using a separate instance, but I tried to upload the editor cache to the cache server.

However, I still can’t get reliably over the issue with UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData.Run resulting in an ArgumentException because of an empty path (BuildCache.GetCacheEntry); it seems that some asset has dependency, which doesn’t exist already, and the AssetDatabase.GUIDToAssetPath then returns empty path?

Is there a way to determine (ideally outside of the build process) which assets can have invalid dependencies, so that we can fix them? I tried checking for non-existing MonoBehaviours, but those are fixed already.

[Ryanc_unity]

@pavelkouril

I was using a separate instance, but I tried to upload the editor cache to the cache server.

However, I still can’t get reliably over the issue with UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData.Run resulting in an ArgumentException because of an empty path (BuildCache.GetCacheEntry); it seems that some asset has dependency, which doesn’t exist already, and the AssetDatabase.GUIDToAssetPath then returns empty path?

Is there a way to determine (ideally outside of the build process) which assets can have invalid dependencies, so that we can fix them? I tried checking for non-existing MonoBehaviours, but those are fixed already.

Hmm, honestly sounds like a edge case I should just fix as Assets (or references in this case) can disappear, but the backing data still can be valid as you could restore the asset and the reference will connect back up without any changes. In this case though it sounds like something might be automatically creating objects during the build process incorrectly? I’m not should though as I haven’t run into this locally and been able to debug. PM sent with more info.

On SBP / CS side, how are you uploading the editor cache to the cache server? I take it you wrote your own upload script?

[pavelkouril]

@Ryanc_unity

Hmm, honestly sounds like a edge case I should just fix as Assets (or references in this case) can disappear, but the backing data still can be valid as you could restore the asset and the reference will connect back up without any changes. In this case though it sounds like something might be automatically creating objects during the build process incorrectly? I’m not should though as I haven’t run into this locally and been able to debug. PM sent with more info.

On SBP / CS side, how are you uploading the editor cache to the cache server? I take it you wrote your own upload script?

We uploaded the data using the dialog that appeared in Unity after installing SBP, “Assets/Cache Server/Upload All Assets”

Is it also possible that e.g. some Image component has missing texture as a reference (that was just a wrongly deleted file), and the GUID is still in the meta of the asset, but it doesn’t exist anymore? We also create some files during the build process, but I don’t think any of them are added as dependencies of scenes in any fashion.

[Ryanc_unity]

@pavelkouril

We uploaded the data using the dialog that appeared in Unity after installing SBP, “Assets/Cache Server/Upload All Assets”

Ahh, “Assets/Cache Server/Upload All Assets” is for Asset artifacts, not Build artifacts.

@pavelkouril

Is it also possible that e.g. some Image component has missing texture as a reference (that was just a wrongly deleted file), and the GUID is still in the meta of the asset, but it doesn’t exist anymore? We also create some files during the build process, but I don’t think any of them are added as dependencies of scenes in any fashion.

Hmm, it’s possible. I don’t remember specifically testing that case, and if it is returning that object as a dependency, that’s wrong and I need to fix the “ContentBuildInterface.PrepareScene” API to not do that (also ContentBuildInterface.GetPlayerDependenciesForObjects).

[pavelkouril]

@Ryanc_unity

Hmm, it’s possible. I don’t remember specifically testing that case, and if it is returning that object as a dependency, that’s wrong and I need to fix the “ContentBuildInterface.PrepareScene” API to not do that (also ContentBuildInterface.GetPlayerDependenciesForObjects).

I’ll could try to filter the results of ContentBuildInterface.GetPlayerDependenciesForObjects on my side in SBP tomorrow to those where AssetDatabase.GUIDToAssetPath actually returns existing non-empty path and write the result if it helped with this issue?

[pavelkouril]

@Ryanc_unity

Ahh, “Assets/Cache Server/Upload All Assets” is for Asset artifacts, not Build artifacts.

Will the CS get the build artifacts from the build machine while I build with it for the first time? Or without doing the custom upload script, the server won’t contain the artifacts at all, even during subsequent run?

[Ryanc_unity]

@pavelkouril

Will the CS get the build artifacts from the build machine while I build with it for the first time? Or without doing the custom upload script, the server won’t contain the artifacts at all, even during subsequent run?

SBP uploads artifacts when it generates them as some artifacts need to (currently) follow a Least Recently Used cache pattern while others don’t. So first run it will store a local copy of the artifacts, and upload a copy to the cache server (This is if you set the CacheServerHost and CacheServerPort values on the passed in BundleBuildParameters (or BuildParameters) object.

[Ryanc_unity]

@pavelkouril

I’ll could try to filter the results of ContentBuildInterface.GetPlayerDependenciesForObjects on my side in SBP tomorrow to those where AssetDatabase.GUIDToAssetPath actually returns existing non-empty path and write the result if it helped with this issue?

Ya, filtering those results should work. You would only want to filter the ObjectIdentifier out if guid.Empty() is false and if the path returned is null or empty. As that should indicate missing reference, so save to skip skip.

[pavelkouril]

@Ryanc_unity

Ya, filtering those results should work. You would only want to filter the ObjectIdentifier out if guid.Empty() is false and if the path returned is null or empty. As that should indicate missing reference, so save to skip skip.

I added a filter condition ```
Where(x => x.guid.Empty() || (!x.guid.Empty() && AssetDatabase.GUIDToAssetPath(x.guid.ToString()) != string.Empty && AssetDatabase.GUIDToAssetPath(x.guid.ToString()) != null))

So either I'm doing the filtering wrong, or there is still some other source for dependencies that have empty path. :/

[Ryanc_unity]

Try this:
BuildCache.cs Line 92, add Debug.Log(entry); to the beginning of that function and run it again with the console open, cleared, and collapse turned off. Then look at the entry log right before the error and paste that Log(entry) output and the full stack trace of the error to me.

[pavelkouril]

{51f772310d7ea1e4d9e26e9cca2497af, c6d57e38e20f24411f5b94918cea3dd8}
UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
UnityEngine.Logger:Log(LogType, Object)
UnityEngine.Debug:Log(Object)
UnityEditor.Build.Pipeline.Utilities.BuildCache:GetUpdatedCacheEntry(CacheEntry) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:101)
UnityEditor.Build.Pipeline.Utilities.BuildCache:NeedsRebuild(CachedInfo) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:118)
UnityEditor.Build.Pipeline.Utilities.BuildCache:LoadCachedData(IList`1, IList`1&) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:260)
UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData:Run() (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Tasks\CalculateSceneDependencyData.cs:57)
UnityEditor.Build.Pipeline.BuildTasksRunner:Run(IList`1, IBuildContext) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Shared\BuildTasksRunner.cs:50)
UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&, IList`1, IContextObject[]) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:108)
UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:35)
UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles_Internal(String, IBundleBuildContent, BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:88)
UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles(String, AssetBundleBuild[], BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:54)
foo.editor.<ExecuteBundleBuild>c__AnonStorey2:<>m__0() (at Assets\Editor\Packaging\BuildBundles.cs:804)
BuildPipelineEx.BuildContext:ExecuteWithBuildPlayerSettings(Func`1) (at Assets\Plugins\Editor\BuildPipelineEx\BuildContext.cs:603)
foo.editor.BuildBundles:ExecuteBundleBuild(BuildContext, String, AssetBundleBuild[]) (at Assets\Editor\Packaging\BuildBundles.cs:804)
foo.editor.BuildBundles:smile:oExecuteBuild(BuildContext, AssetBundleSetBuildHelper, String, String, String) (at Assets\Editor\Packaging\BuildBundles.cs:248)
foo.editor.FinalizeBuildStep:Execute(BuildContext) (at Assets\Editor\Packaging\BuildBundles.cs:118)
BuildPipelineEx.CommandLineBuildBridge:ExecuteCustomBuildSteps(BuildContext, BuildPhase) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:118)
BuildPipelineEx.CommandLineBuildBridge:PostBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:635)
BuildPipelineEx.CommandLineBuildBridge:smile:oRunBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:597)
BuildPipelineEx.CommandLineBuildBridge:RunBuild() (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:510)
(Filename: Assets/Editor/Packaging/BuildBundles.cs Line: 804)

ArgumentException: Path is empty
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x00027] in <f2e6809acb14476a81f399aeb800f8f2>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in <f2e6809acb14476a81f399aeb800f8f2>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)
  at UnityEditor.Build.Pipeline.Utilities.HashingMethods.CalculateFile (System.String filePath) [0x00004] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\HashingMethods.cs:294
  at UnityEditor.Build.Pipeline.Utilities.BuildCache.GetCacheEntry (UnityEditor.GUID asset) [0x000bc] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:68
  at UnityEditor.Build.Pipeline.Utilities.BuildCache.GetUpdatedCacheEntry (UnityEditor.Build.Pipeline.Utilities.CacheEntry entry) [0x00040] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:106
  at UnityEditor.Build.Pipeline.Utilities.BuildCache.NeedsRebuild (UnityEditor.Build.Pipeline.Utilities.CachedInfo info) [0x0006b] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:118
  at UnityEditor.Build.Pipeline.Utilities.BuildCache.LoadCachedData (System.Collections.Generic.IList`1[T] entries, System.Collections.Generic.IList`1[UnityEditor.Build.Pipeline.Utilities.CachedInfo]& cachedInfos) [0x00175] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildCache.cs:260
  at UnityEditor.Build.Pipeline.Tasks.CalculateSceneDependencyData.Run () [0x0006e] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Tasks\CalculateSceneDependencyData.cs:57
  at UnityEditor.Build.Pipeline.BuildTasksRunner.Run (System.Collections.Generic.IList`1[T] pipeline, UnityEditor.Build.Pipeline.Interfaces.IBuildContext context) [0x00097] in C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Shared\BuildTasksRunner.cs:50
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
UnityEditor.Build.Pipeline.Utilities.BuildLogger:LogException(Exception) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Utilities\BuildLogger.cs:77)
UnityEditor.Build.Pipeline.BuildTasksRunner:Run(IList`1, IBuildContext) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\Shared\BuildTasksRunner.cs:57)
UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&, IList`1, IContextObject[]) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:108)
UnityEditor.Build.Pipeline.ContentPipeline:BuildAssetBundles(IBundleBuildParameters, IBundleBuildContent, IBundleBuildResults&) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\ContentPipeline.cs:35)
UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles_Internal(String, IBundleBuildContent, BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:88)
UnityEditor.Build.Pipeline.LegacyBuildPipeline:BuildAssetBundles(String, AssetBundleBuild[], BuildAssetBundleOptions, BuildTarget) (at C:\Users\kourilpav.BISTUDIO\AppData\Local\Unity\cache\packages\packages.unity.com\com.unity.scriptablebuildpipeline@1.1.1-preview\Editor\LegacyBuildPipeline.cs:54)
foo.editor.<ExecuteBundleBuild>c__AnonStorey2:<>m__0() (at Assets\Editor\Packaging\BuildBundles.cs:804)
BuildPipelineEx.BuildContext:ExecuteWithBuildPlayerSettings(Func`1) (at Assets\Plugins\Editor\BuildPipelineEx\BuildContext.cs:603)
foo.editor.BuildBundles:ExecuteBundleBuild(BuildContext, String, AssetBundleBuild[]) (at Assets\Editor\Packaging\BuildBundles.cs:804)
foo.editor.BuildBundles:smile:oExecuteBuild(BuildContext, AssetBundleSetBuildHelper, String, String, String) (at Assets\Editor\Packaging\BuildBundles.cs:248)
foo.editor.FinalizeBuildStep:Execute(BuildContext) (at Assets\Editor\Packaging\BuildBundles.cs:118)
BuildPipelineEx.CommandLineBuildBridge:ExecuteCustomBuildSteps(BuildContext, BuildPhase) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:118)
BuildPipelineEx.CommandLineBuildBridge:PostBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:635)
BuildPipelineEx.CommandLineBuildBridge:smile:oRunBuild(BuildContext) (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:597)
BuildPipelineEx.CommandLineBuildBridge:RunBuild() (at Assets\Plugins\Editor\BuildPipelineEx\CommandLineBuildBridge.cs:510)
(Filename: <f2e6809acb14476a81f399aeb800f8f2> Line: 0)

I tried searching for the
51f772310d7ea1e4d9e26e9cca2497af and
c6d57e38e20f24411f5b94918cea3dd8. The 51f772310d7ea1e4d9e26e9cca2497af was a reference to a Missing (Audio Clip) on a prefab, but even when I removed these references (can’t find a single file having this string in the Assets folder), I got the same error. Or are these asset dependencies stored somehow cached and I will need to evict that cache too?

[Ryanc_unity]

Oh! That error makes a lot of sense. It’s trying to load data from the cache, because the entry exists, but the file in the cache is not found. So something failed to get added to the cache on the first build.

Purge the build cache, run the build again and watch for errors. It’s likely I missed cleanup in BuildCache.cs on move failures in an edge case.

[pavelkouril]

@Ryanc_unity

Oh! That error makes a lot of sense. It’s trying to load data from the cache, because the entry exists, but the file in the cache is not found. So something failed to get added to the cache on the first build.

Purge the build cache, run the build again and watch for errors. It’s likely I missed cleanup in BuildCache.cs on move failures in an edge case.

I’ll try it tomorrow them; by purging the build cache, you mean something in Library folder, or just the contents of the cache server (if there are any).

[Ryanc_unity]

@pavelkouril

I’ll try it tomorrow them; by purging the build cache, you mean something in Library folder, or just the contents of the cache server (if there are any).

“Library/BuildCache” is the local copy of the build cache. Purging, is just deleting that folder. There is also a menu item for this: “Window > Asset Management > Purge Build Cache”.

[pavelkouril]

Ok, thanks!

We were finally able to build using SBP. Shame there was no speedup for builds when using SBP+Cache Server for us, so we will probably wait until the package goes out of preview (hopefully there will be less edge cases by then?).

[Ryanc_unity]

SBP still gives you local speed ups on subsequent builds. Cache Server is just if you wish to share the build cache from one machine with other machines. Since you were able to get it working using just SBP, I would purge the Cache Server you were using for this and reconnect SBP to the Cache server and try again. Don’t use “Assets/Cache Server/Upload All Assets” and make sure this cache server is ONLY connected to SBP via the BundleBuildParameters input class and not the Editor via Editor Preferences.