Accelerator is caching an empty/corrupt asset.

Unity Engine
1

Hi,

We have just experienced a problem where a broken asset (0 bytes long) got uploaded to the accelerator and now all builds fail, with this error in the log:
Error loading the file 'Library/Artifacts/32/326e47e820bc46df02a2958b7dd90697'. File is either empty or corrupted, please verify the file contents.

From the accelarator logs I can see that one of our build slaves uploaded the 0 sized asset (utc timestamps):

{"level":"debug","ts":"2020-10-23T22:18:27.919Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11482,"remote":"192.168.16.140:64500","segment_id":24838,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":3818424}
**{"level":"debug","ts":"2020-10-23T22:19:32.878Z","msg":"handled putRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11480,"remote":"192.168.16.140:64499","segment_id":26128,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","size":0,"status":"ok","status_code":0} << BAD UPLOAD**
{"level":"debug","ts":"2020-10-23T22:34:18.813Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11488,"remote":"192.168.16.141:63744","segment_id":7224,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":0}
{"level":"debug","ts":"2020-10-23T22:44:45.715Z","msg":"handled getRequest","agent_id":"teamcity_id","agent_name":"teamcity","component":"pbservice","subprocess_id":6,"conn_id":11490,"remote":"192.168.16.140:52390","segment_id":7224,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"326e47e820bc46df02a2958b7dd90697","status":"ok","status_code":0,"size":0}

This is the log from the Unity that uploaded the corrupted asset (utc+2h timestamps):

[00:19:19] : [Post Processing Catalog Entries]
[00:19:19] : [Post Processing Catalog Entries] TPImporter: OnPreprocessTexture(Assets/Sprites/Characters/Atlases/Low/Otho_Armored.png)
[00:19:19] : [Post Processing Catalog Entries] TPImporter: sheet exists in db, no changes
[00:19:19] : [Post Processing Catalog Entries] TPImporter: Successfully loaded tpsheet file containing 370 sprite definitions
[00:19:19] : [Post Processing Catalog Entries] TPImporter: set texture type + mesh type
[00:19:19] : [Post Processing Catalog Entries] TPImporter: meta data hasn't changed
[00:19:19] : [Post Processing Catalog Entries]
[00:19:23] : [Post Processing Catalog Entries] TPImporter: Texture scale factor: 1
[00:19:23] : [Post Processing Catalog Entries] TPImporter: Updating geometry of 370 sprites
[00:19:23] : [Post Processing Catalog Entries]
[00:19:33] : [Post Processing Catalog Entries] Done importing asset: 'Assets/Sprites/Characters/Atlases/Low/Otho_Armored.png' (target hash: '812098c51feb6381628b111786045e53') in 13.586947 seconds
[00:19:33] : [Post Processing Catalog Entries] RemoteAssetCache::AddArtifactToCacheServer - artifactKey='Guid(e3f006035ddb0de4e82479b8ed477cdf) Importer(-1,00000000000000000000000000000000)' Target hash='812098c51feb6381628b111786045e53' Key (and library path)='Library/Artifacts/32/326e47e820bc46df02a2958b7dd90697'
[00:19:33] : [Post Processing Catalog Entries] RemoteAssetCache - Upload - Artifact - success:true, namespace:81e94844d19a16919208533e08183531, key:326e47e820bc46df02a2958b7dd90697
[00:19:33] : [Post Processing Catalog Entries] RemoteAssetCache::AddArtifactToCacheServer - artifactKey='Guid(e3f006035ddb0de4e82479b8ed477cdf) Importer(-1,00000000000000000000000000000000)' Target hash='812098c51feb6381628b111786045e53' Key (and library path)='Library/Artifacts/c4/c48e12a7e1eb5bf520e3f4b4b7fa076b'

This is the error we get when trying to build:

[23:40:50] : [Unity Package Manager] UnityEditor.AssetDatabase:GetMainAssetTypeAtPath(String)
[23:40:50] : [Unity Package Manager] UnityEditor.Localization.LocalizationAssetPostProcessor:OnPostprocessAllAssets(String[ ], String[ ], String[ ], String[ ]) (at Assets\com.unity.localization@0.8.0-preview\Editor\Asset Pipeline\LocalizationAssetPostProcessor.cs:35)
[23:40:50] : [Unity Package Manager] System.Reflection.MonoMethod:InternalInvoke(Object, Object[ ], Exception&)
[23:40:50] : [Unity Package Manager] System.Reflection.MonoMethod:Invoke(Object, BindingFlags, Binder, Object[ ], CultureInfo)
[23:40:50] : [Unity Package Manager] System.Reflection.MethodBase:Invoke(Object, Object[ ])
[23:40:50] : [Unity Package Manager] UnityEditor.AssetPostprocessingInternal:InvokeMethod(MethodInfo, Object[ ])
[23:40:50] : [Unity Package Manager] UnityEditor.AssetPostprocessingInternal:PostprocessAllAssets(String[ ], String[ ], String[ ], String[ ], String[ ])

We don’t know how to fix this problem, if we delete the broken assets from the cache, a different Unity instance will upload the broken version again. We also tried deleting the entry, and do a force reimport in the hopes that a fixed version would get uploaded, but it appeared like a broken version got uploaded.
./bin/unity-accelerator cache delete 127.0.0.1:10080 78b2fdc254b0a4f8a4693a81bb96598a 326e47e820bc46df02a2958b7dd90697

This time around we could fix it by just changing the guild of the asset (as there were only a few references to the broken asset), but that might not be possible next time, if the asset is referencved from all over.
Please advice on how this should handled.

Unity: 2019.4.8f1
Accelerator: v1.0.604+g695e221

1 Like
2

Hi, is there a particular asset or type that this happens for you guys? We don’t have any recommended way of handling this currently, as we not exactly sure what would be causing this.

When ‘Assets/Sprites/Characters/Atlases/Low/Otho_Armored.png’ has been imported, is the import result also 0 byte? Wondering if it’s the uploading or the importing that goes wrong. Because the accelerator log seems to indicate that it just receives 0 bytes.

3

I can’t tell if the imported asset was also 0 bytes long on the build machine, that build have been cleaned since then.
I found a few other errors in the log of build taht pushed the 0 bytes long asset to the accelerator:

A few (around 10) of these:

[00:21:46] : [Post Processing Catalog Entries] UnityEditor.AssetImporter:GetAtPath(String)
[00:21:46] : [Post Processing Catalog Entries] UnityEditor.ModelImporterPostProcessor:OnPostprocessAllAssets(String[ ], String[ ], String[ ], String[ ])
[00:21:46] : [Post Processing Catalog Entries] System.Reflection.MonoMethod:InternalInvoke(Object, Object[ ], Exception&)
[00:21:46] : [Post Processing Catalog Entries] System.Reflection.MonoMethod:Invoke(Object, BindingFlags, Binder, Object[ ], CultureInfo)
[00:21:46] : [Post Processing Catalog Entries] System.Reflection.MethodBase:Invoke(Object, Object[ ])
[00:21:46] : [Post Processing Catalog Entries] UnityEditor.AssetPostprocessingInternal:InvokeMethod(MethodInfo, Object[ ])
[00:21:46] : [Post Processing Catalog Entries] UnityEditor.AssetPostprocessingInternal:PostprocessAllAssets(String[ ], String[ ], String[ ], String[ ], String[ ])
[00:21:46] : [Post Processing Catalog Entries] UnityEditor.AssetDatabase:Refresh(ImportAssetOptions)
[00:21:46] : [Post Processing Catalog Entries] UnityEditor.AssetDatabase:Refresh()
[00:21:46] : [Post Processing Catalog Entries] GooglePlayGames.Editor.GPGSAndroidSetupUI:PerformSetup(String, String, String) (at Assets\GooglePlayGames\Editor\GPGSAndroidSetupUI.cs:200)
[00:21:46] : [Post Processing Catalog Entries] Build:BuildAndroid(Boolean, Boolean, Boolean, Boolean, Boolean, Boolean) (at Assets\Editor\Build\BuildAndroid.cs:148)
[00:21:46] : [Post Processing Catalog Entries] Build:BuildAndroidDev(Boolean, Boolean, Boolean, Boolean) (at Assets\Editor\Build\BuildAndroid.cs:80)
[00:21:46] : [Post Processing Catalog Entries] Build:BuildAndroidDev() (at Assets\Editor\Build\BuildAndroid.cs:41)
[00:21:46] : [Post Processing Catalog Entries]
[00:21:46] : [Post Processing Catalog Entries] [ line 0]
[00:21:46] : [Post Processing Catalog Entries] (Filename: Assets/GooglePlayGames/Editor/GPGSAndroidSetupUI.cs Line: 200)

This was a clean build, so the Library folder did not exist at the start of the build, so it’s more likely that the file was coprrupted than it was from a different version of Unity.

A few assertions of and a abort:

[00:24:57] : [Building Player] (Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 453)
[00:24:57] : [Building Player]
[00:24:57] : [Building Player] Assertion failed on expression: 'success && actual == size'
[00:24:57] : [Building Player] (Filename: C:\buildslave\unity\build\Runtime/Utilities/FileVFS.cpp Line: 462)
[00:24:57] : [Building Player]
[00:24:57] : [Building Player]
[00:24:57] : [Building Player]
[00:24:57] : [Building Player] Aborting batchmode due to failure:
[00:24:57] : [Building Player] Fatal Error! dataOffset != file.GetPosition()

I can’t find any messages in the log about running out of disk space, but the errors kinda leads me to believe that it could have happend, and Unity did not react to failed writes?

1 Like
4

I think your conclusion sounds about right. Both the assertions you mention are from the Virtual File System layer when writing to disk.This is normally the error you would get when running out of disk space. The message could be clearer though :wink:

However, running out of disk space should not result in uploading a 0-byte file, and thereby polluting other users of the accelerator. So we will look into a fix for this case.

1 Like
5

Did you find a fix for this case? :slight_smile:

This error still appears rarely but it does.

1 Like