Accelerator v1.0.130+g5e61869 doesn't DO anything? :)

Unity Engine
1

After some fiddling with configs (despite Unity 2019.3.0f1 insisting “connection works”), I got accelerator to run on a separate Linux machine, on ports that it could actually use. I have been seeing the same behaviour with a local accelerator install unter Windows (on two separate machines, in two separate organizations), though.

I set this up to only accelerate the V2 asset pipeline (no Collaborate shenanigans)

Problem 1: There seems zero discernible acceleration of Unity Imports. There seems only a very low utilization of disk space on the cache server. (sometimes local installs show a bit of usage)

Problem 2: When in Unity I select “Assets->Cache Server->Upload Library to Cache Server”, unity gets “busy” (and is doing SOMETHING because when I disconnect it from the net or kill the accelerator process, it sometimes drops out of this stage) - but doesn’t respond to input anymore, for hours. There seems no discernible process, and only homeopathic CPU and Network usage.

On the accelerator side, whatever I do, all I see in http://:api/agent-health is: {“version”:“v1.0.130+g5e61869”,“up”:“3m22.035853273s”,“load”:0}

Load is always zero. It appears the only actual log entries (which get flushed to disk inconveniently only if I terminate accelerator), are from the connection tests in Unity.

Thanks in advance for your help. :slight_smile:

2

12 minutes into one of these “Uploading” sessions, this is what my cachedb looks like:

<user>@<myserver>:~/.config/unity-accelerator/cachedb$ du -h
8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6/e4
8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6/29
20K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6
12K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/5d
8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/4c
12K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/f8
36K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24
60K     ./e24b6d24ba7d7918e08be3d9b15d5fd8
92K     .

I must kill unity to ever get it back.

But it DOES connect.

TCP    192.168.0.190:54410    192.168.0.196:10245    ESTABLISHED
3 
{"level":"info","ts":"2020-02-13T16:13:26+01:00","msg":"tool wrun start","cmd":"/opt/Unity/accelerator/unity-accelerator tool wrun --persist /home/berlin/.config/unity-accelerator","pid":4014}
{"level":"info","ts":"2020-02-13T16:13:26.719+0100","msg":"sysinfo","agent_id":"neptune_id","agent_name":"neptune","component":"agent","pid":4014,"agentversion":"v1.0.130+g5e61869","goversion":"go1.13.1","os":"linux","arch":"amd64","compiler":"gc","maxprocs":8,"numcpu":8}
{"level":"info","ts":"2020-02-13T16:13:27.129+0100","msg":"handling connections","agent_id":"neptune_id","agent_name":"neptune","component":"pbservice","subprocess_id":7,"url":"agentprotobuf://neptune:10245","addr":"[::]:10245","using_tls":false,"auth_required":false}
{"level":"info","ts":"2020-02-13T16:13:27.129+0100","msg":"handling connections","agent_id":"neptune_id","agent_name":"neptune","component":"httpservice","subprocess_id":6,"url":"http://neptune:10000","addr":"[::]:10000"}

Some logs

4

I found out that the /etc/init.d/unity-accelerator script leaks processes when you start it repeatedly by accident.

Some of the processes even get stuck (need to be killed -9), but not all of them.

This is usually the case when your logs show that it couldn’t bind to an address.

5

There are embedded tools in the accelerator we can use to try to debug.

You can try to put something into the cache with:
unity-accelerator tool put neptune:10245 namespace key

Like many Unix tools, no output equals success here.

Then, you can try to get that item from the cache with:
unity-accelerator tool get neptune:10245 namespace key

A longer running test may be done with:
unity-accelerator tool perf neptune:10245

If these all run well, perhaps we can narrow the problem down to the project on the Unity Editor side of things.

You may also find doing an HTTP request to your http://neptune:10000/metrics to be useful.

6

Thanks for these tips.

The unity-accelerator tool works great locally on the linux server, as well as remotely on one of the machines I’ve been using Unity Editor on (Windows).

I’ve tried PUTting and GETting a few files, works as expected.

The perf test seems to work as well (this is on the remote machine, across the network):

10000 ops of 100% put and 0% get with 131072 bytes each, 25.29s total, 395.39 r/s, 51824403.70 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 100% put and 0% get with 131072 bytes each, 24.88s total, 401.87 r/s, 52674293.60 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 0% put and 100% get with 131072 bytes each, 0.83s total, 12061.62 r/s, 1580940900.59 b/s, 0 hits, 10000 misses, 0 put errors, 0 get errors
10000 ops of 0% put and 100% get with 131072 bytes each, 29.74s total, 336.23 r/s, 44069834.71 b/s, 10000 hits, 0 misses, 0 put errors, 0 get errors
(still running)

This also results in some load. (does 1 mean 1 as in 1.0? the machine isn’t breaking a sweat but I guess this is mostly throughput bottlenecked, I haven’t hooked up the 2nd ethernet interface)

{"version":"v1.0.130+g5e61869","up":"17m37.079119876s","load":1}

Running the perf test locally does heat up the server :slight_smile:

10000 ops of 100% put and 0% get with 131072 bytes each, 1.21s total, 8286.52 r/s, 1086130250.69 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 100% put and 0% get with 131072 bytes each, 1.37s total, 7307.67 r/s, 957830931.29 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 0% put and 100% get with 131072 bytes each, 0.24s total, 42085.17 r/s, 5516187473.76 b/s, 0 hits, 10000 misses, 0 put errors, 0 get errors
10000 ops of 0% put and 100% get with 131072 bytes each, 1.35s total, 7429.80 r/s, 973838190.52 b/s, 10000 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 25% put and 75% get with 131072 bytes each, 1.42s total, 7062.89 r/s, 925746733.94 b/s, 7500 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 50% put and 50% get with 131072 bytes each, 1.48s total, 6741.65 r/s, 883641359.84 b/s, 5000 hits, 0 misses, 0 put errors, 0 get errors
10000 ops of 75% put and 25% get with 131072 bytes each, 1.51s total, 6608.37 r/s, 866172077.60 b/s, 2500 hits, 0 misses, 0 put errors, 0 get errors

 {"version":"v1.0.130+g5e61869","up":"21m41.849513351s","load":2}
7

Coming straight out of these tests, using Unity 2019.3.0f1, it locks up exactly the same.

I also see this “bug” occasionally in Project settings:

5476944--560460--upload_2020-2-13_17-15-10.png
5476944--560457--upload_2020-2-13_17-14-23.png

(global settings are of course successfully connecting in Preferences->Editor)

8

Now what’s interesting is… my other project (I picked one that’s very unrelated to the ones that weren’t working) … doesn’t even have the Assets->Asset Server menu. It IS a project from a different organization though.

9

You may need to upgrade your project to use the asset pipeline v2 on the other project you’re testing on.

In the /metrics data you posted, I noticed it says you have uta_agent_cache_bytes_stored 1.333615737e+09 which is about 1.25GB and uta_agent_cache_bytes_in 3.955045843e+09 which is about 3.7GB transfers into the accelerator. But I’m guessing a du -sh of your cachedb doesn’t show ~1.25GB? These values are also reflected in the protobuf_cache metrics, which indicates use of the asset pipeline v2.

My boss here indicated you might check the Editor logs similarly to the thread here Are we doing this right? to see if the Editor is actually connecting and using the accelerator.

You can also turn on debug logging on the accelerator side if you wish, by editing the unity-accelerator.cfg file and flipping the Debug value to true and restarting the accelerator. With debug on, every request will be logged.

10

Yes these values came from the tests, the cachedb was practically empty before.

The project IS asset pipeline v2.

5477073--560478--upload_2020-2-13_17-37-34.png

Unless there is another concurrent setting?

11

My editor logs are practically empty. (there are two collab-accelerator related lines).

[collab-accelerator] discovery started due to a new cloud project binding
UPID Received '36a28678-5c67-4da9-b25f-24ff20d2972f'.

.....

[collab-accelerator] Set collab endpoint to https://collab.cloud.unity3d.com (collab-service)
<RI> Initialized touch support.

When I try to connect to a non-existent server, the error shows up in the editor log.

When I connect to my existing (and now tested…) accelerator server, only one thing appears in the logs:

Refresh completed in 0.156207 seconds.
RefreshInfo: RefreshV2(ForceSynchronousImport)
RefreshProfiler: Total: 155.487ms
    InvokeBeforeRefreshCallbacks: 0.522ms
    ApplyChangesToAssetFolders: 0.096ms
    WriteModifiedImportersToTextMetaFiles: 0.000ms
    CleanLegacyArtifacts: 0.000ms
    Scan: 138.931ms
    OnSourceAssetsModified: 0.000ms
    UnregisterDeletedAssets: 0.000ms
    InitializeImportedAssetsSnapshot: 8.737ms
    GetAllGuidsForCategorization: 0.000ms
    CategorizeAssets: 0.000ms
    ImportAndPostprocessOutOfDateAssets: 0.000ms (0.001ms without children)
        ImportManagerImport: 0.000ms (0.000ms without children)
            ImportInProcess: 0.000ms
            ImportOutOfProcess: 0.000ms
            UpdateCategorizedAssets: 0.000ms
            RemoteAssetCacheGetArtifact: 0.000ms (0.000ms without children)
                RemoteAssetCacheResolve: 0.000ms
                RemoteAssetCacheDownloadFile: 0.000ms
        CompileScripts: 0.000ms
        PostProcessAllAssets: 0.000ms
        ReloadImportedAssets: 0.000ms
        VerifyAssetsAreUpToDateAndCorrect: 0.000ms
        EnsureUptoDateAssetsAreRegisteredWithGuidPM: 0.000ms
        InitializingProgressBar: 0.000ms
        PostProcessAllAssetNotificationsAddChangedAssets: 0.000ms
        OnDemandSchedulerStart: 0.000ms
        RestoreLoadedAssetsState: 0.000ms
    InvokeProjectHasChanged: 0.000ms
    UpdateImportedAssetsSnapshot: -0.001ms
    ReloadSourceAssets: 0.000ms
    UnloadImportedAssets: 0.000ms
    Hotreload: 0.112ms
    FixTempGuids: 0.003ms
    VerifyGuidPMRegistrations: 0.000ms
    GatherAllCurrentPrimaryArtifactRevisions: 0.755ms
    UnloadStreamsBegin: 0.086ms
    LoadedImportedAssetsSnapshotReleaseGCHandles: 1.715ms
    GetLoadedSourceAssetsSnapshot: 1.891ms
    PersistCurrentRevisions: 0.000ms
    UnloadStreamsEnd: 0.029ms
    Untracked: 2.610ms

Unity then freezes (doesn’t respond to input, but windows doesn’t think the process is unresponsive)

12

Ok, creating a new project (3D with Extras template) in 2019.3.1f1 DOES utilize the server upon import.

13

And now the import of the other project works.

But I still need to find the “Cache Server->Upload All Assets” tool item and kill it. I suspect it’s from the previous Cache Server, but I’ve been grepping the entire project and all packages and haven’t found it yet.

14

Oh good, I was actually about to type that up – I’d just tried a fresh project over here too to double-check what you should see in the logs on both ends for that. Menu Assets->Reimport All – on a new tester project - should definitely hit the cache server right at the end of its reimport, just before the Editor is ready to use again. If you have debug on on the accelerator service you should see a flood of requests, mostly getRequests.

I will see if I can find someone to help out with your existing project that is having issues – I think I might be beyond my knowledge levels there. :slight_smile:

15 
C:\Projects\tintin\tintin-unity\Library\PackageCache\com.unity.scriptablebuildpipeline@1.5.4\Editor\CacheServer\CacheServerUploaderWindow.cs:
   52:                     "This will upload all assets in your Library folder to the specified Cache Server.", "Continue", "Cancel"))
   81:         [MenuItem("Assets/Cache Server/Upload All Assets")]

The delinquent Asset Menu Entry is from this Unity package. I don’t think this works with Accelerator.

5477217--560526--upload_2020-2-13_18-8-15.png
5477217--560526--upload_2020-2-13_18-8-15.png482×508 26.4 KB

16

My problem appears to be solved, thank you for showing me the “tool” subcommand in Accelerator, I’m already thinking of fun ways to abuse this :smile:, err, I mean, troubleshoot in the future.

Summary:

  • Scriptable Build Pipeline provides an Asset Upload command that doesn’t work.
  • Reimport/Deleting Library works acceptably (our project has maybe a 10-15 min import time)
  • The earlier Import problems were probably from multiple Accelerator installs / processes occupying the same resources or something.
1 Like