(CASE IN-7885) [0.51] Linux server build runtime not working properly

Build with Unity 2021.3.4f1. Get the following error when launching:

“memorysetup-cache-allocator-block-size=4194304” “memorysetup-typetree-allocator-block-size=2097152” “memorysetup-profiler-bucket-allocator-granularity=16” “memorysetup-profiler-bucket-allocator-bucket-count=8” “memorysetup-profiler-bucket-allocator-block-size=4194304” “memorysetup-profiler-bucket-allocator-block-count=1” “memorysetup-profiler-allocator-block-size=16777216” “memorysetup-profiler-editor-allocator-block-size=1048576” “memorysetup-temp-allocator-size-main=4194304” “memorysetup-job-temp-allocator-block-size=2097152” “memorysetup-job-temp-allocator-block-size-background=1048576” “memorysetup-job-temp-allocator-reduction-small-platforms=262144” “memorysetup-temp-allocator-size-background-worker=32768” “memorysetup-temp-allocator-size-job-worker=262144” “memorysetup-temp-allocator-size-preload-manager=262144” “memorysetup-temp-allocator-size-nav-mesh-worker=65536” “memorysetup-temp-allocator-size-audio-worker=65536” “memorysetup-temp-allocator-size-cloud-worker=32768” “memorysetup-temp-allocator-size-gfx=262144” error: XDG_RUNTIME_DIR not set in the environment.

thanks for reporting. Not sure is netcode specific though. Worth posting this also in the dots forum for visibility.

When I start the dedicated server runtime, it just eat up CPU and memory to max and then crash.

Again, not sure is because of the NetCode package that this is happening. Can be almost everything.
Only on Linux or also other platform?

Still testing. But looks like most probably linux platform at Unity 2021.3.4f1 not able to get a working build. Both Window, Mac, Linux and Dedicated Server for linux build are not working.

8212605--1072071--upload_2022-6-17_15-2-20.png

ok, wait a minute: why are you building using the Unity compilation pipeline and not using the scriptable build pipeline (and so build config) ?
We are not supporting building the dedicated server target in dots (and NetCode more specifically) in 0.51. You should still use the ScriptableBuildPipeline for building the server.
I’m missing something?

Ya. I’m still using dots build configuration to build linux server. At Unity 2021.3.4f1 it’s now completely not working anymore (Refer to #1 post). Now I’m using Unity 2021.3.4f1 to build client and Unity 2020.3.35 to build linux server.

ok, but why are you targeting the DedicatedServer platform? (as I saw in the other post). I may be just mistaken

Just trying but realize that switch back to PC, Mac & Linux Standalone to build Linux server still get the same result.

I want to clarify that linux server build with dots 0.51 release at Unity 2021.3.4f1 for dots netcode project not working properly but Unity 2020.3.35f1 is working nicely.

Interesting, thanks for the quite important detail.

I also experience continuously increasing memory usage (and CPU) usage on a netcode linux server. Memory increase happens at around 0.2% per second of 1 GB of ram. Even when no clients are connected to the server, CPU and Ram Usage increases steadily.

We are using physics prediction, but otherwise, not many systems are currently running, no entities are created or destroyed per frame (except when the players spawn).

We were building with Unity 2020.3.33f (tried both entities 0.50 and 0.51)

0.2% increase is quite a lot (we are speaking about 2 MB/s …)
If memory increases and your are not allocating anything at all, then we have potentially a leak.
Question being; is only the linux server that does that or also a osx/win one?
Also: are you able, by disabling systems etc etc, to reach a point where the memory and cpu are stable?
That may at least bisect the problem a little bit.
If even an empty server (that just accept connection and does nothing) increase cpu and memory, than the is absolutely a leak or in entities or netcode.

I created the following tests:

Use the getting started tutorial (cube moving with wasd) and created a Linux build running inside a docker.

Unity version: 2020.3.36f1
Netcode version: 0.51

No memory leaks can be observed even when run for longer times.

Also tried to add the predicted physics systems into the mix, and still no memory leaks, so most likely there is no memory leak from the dotsnet package or entities package directly, but something from the systems created by us (or it happened during an upgrade process form 0.50 to 0.51).

Will investigate further.

Unity version: 2021.3.4f1
Netcode version: 0.51

@CMarastoni Tried official netcube project to build linux server runtime and test. It just crash when I run the runtime. Still get similar error log like post #1. From Player log file, it stopped at Unable to load player prefs.

[UnityMemory] Configuration Parameters - Can be set up in boot.config
“memorysetup-bucket-allocator-granularity=16”
“memorysetup-bucket-allocator-bucket-count=8”
“memorysetup-bucket-allocator-block-size=4194304”
“memorysetup-bucket-allocator-block-count=1”
“memorysetup-main-allocator-block-size=16777216”
“memorysetup-thread-allocator-block-size=16777216”
“memorysetup-gfx-main-allocator-block-size=16777216”
“memorysetup-gfx-thread-allocator-block-size=16777216”
“memorysetup-cache-allocator-block-size=4194304”
“memorysetup-typetree-allocator-block-size=2097152”
“memorysetup-profiler-bucket-allocator-granularity=16”
“memorysetup-profiler-bucket-allocator-bucket-count=8”
“memorysetup-profiler-bucket-allocator-block-size=4194304”
“memorysetup-profiler-bucket-allocator-block-count=1”
“memorysetup-profiler-allocator-block-size=16777216”
“memorysetup-profiler-editor-allocator-block-size=1048576”
“memorysetup-temp-allocator-size-main=4194304”
“memorysetup-job-temp-allocator-block-size=2097152”
“memorysetup-job-temp-allocator-block-size-background=1048576”
“memorysetup-job-temp-allocator-reduction-small-platforms=262144”
“memorysetup-temp-allocator-size-background-worker=32768”
“memorysetup-temp-allocator-size-job-worker=262144”
“memorysetup-temp-allocator-size-preload-manager=262144”
“memorysetup-temp-allocator-size-nav-mesh-worker=65536”
“memorysetup-temp-allocator-size-audio-worker=65536”
“memorysetup-temp-allocator-size-cloud-worker=32768”
“memorysetup-temp-allocator-size-gfx=262144”

I also get the player log from .config folder:

Mono linux runtime build

Mono path[0] = ‘/home/ec2-user/sample/sample_Data/Managed’
Mono config path = ‘/home/ec2-user/sample/sample_Data/MonoBleedingEdge/etc’
Preloaded ‘lib_burst_generated.so’
PlayerPrefs - Creating folder: /home/ec2-user/.config/unity3d/sample
Unable to load player prefs
Desktop is 0 x 0 @ 0 Hz

Il2cpp linux runtime build

Preloaded ‘lib_burst_generated.so’
PlayerPrefs - Creating folder: /home/ec2-user/.config/unity3d/sample
Unable to load player prefs
Desktop is 0 x 0 @ 0 Hz

================================================================
For successfully running linux server runtime at Unity 2020.3.35, u will see something like the following:

Preloaded ‘lib_burst_generated.so’
Unable to load player prefs
Initialize engine version: 2020.3.35f1 (18e4db7a9996)
[Subsystems] Discovering subsystems at path /home/ec2-user/sample/sample_Data/Un itySubsystems
Forcing GfxDevice: Null
GfxDevice: creating device client; threaded=0
NullGfxDevice:
Version: NULL 1.0 [1.0]
Renderer: Null Device
Vendor: Unity Technologies
ERROR: Shader Hidden/Universal Render Pipeline/Blit shader is not supported on t his GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/CopyDepth shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/ScreenSpaceShadows shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/Sampling shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/LutBuilderLdr shader is not suppo rted on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/LutBuilderHdr shader is not suppo rted on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/UberPost shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: ‘Hidden/Universal Render Pipeline/StencilDeferred’ - All subshaders removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten tu rning Fallback off?
ERROR: Shader Hidden/Universal Render Pipeline/StencilDeferred shader is not sup ported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: ‘Hidden/Universal Render Pipeline/StencilDeferred’ - All subshaders removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten tu rning Fallback off?
ERROR: Shader Hidden/Universal Render Pipeline/Stop NaN shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/SubpixelMorphologicalAntialiasing shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/GaussianDepthOfField shader is no t supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/BokehDepthOfField shader is not s upported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/CameraMotionBlur shader is not su pported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/PaniniProjection shader is not su pported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/Bloom shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Hidden/Universal Render Pipeline/FinalPost shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Sprites/Default shader is not supported on this GPU (none of subsh aders/fallbacks are suitable)
ERROR: Shader Sprites/Mask shader is not supported on this GPU (none of subshade rs/fallbacks are suitable)
ERROR: Shader Graphy/Graph Mobile shader is not supported on this GPU (none of s ubshaders/fallbacks are suitable)
ERROR: Shader GUI/Text Shader shader is not supported on this GPU (none of subsh aders/fallbacks are suitable)
ERROR: Shader Graphy/Graph Standard shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
UnloadTime: 0.382368 ms
ERROR: Shader UI/Default shader is not supported on this GPU (none of subshaders /fallbacks are suitable)
WARNING: Shader Unsupported: ‘Universal Render Pipeline/Lit’ - All subshaders re moved
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten tu rning Fallback off?
ERROR: Shader Universal Render Pipeline/Lit shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
WARNING: Shader Unsupported: ‘Universal Render Pipeline/Lit’ - All subshaders re moved
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader If subshaders removal was intentional, you may have forgotten tu rning Fallback off?
Streamed scene with 155ms latency from /home/ec2-user/sample/sample_Data/Streami ngAssets/EntityScenes/b3b4427e1963245ed97bcf69eaca92da.0.entities
^CSetting up 1 worker threads for Enlighten.
Thread → id: 7f8600b78700 → priority: 1
Unloading the last loaded scene Assets/Samples/Scenes/NetCube.unity(build index: 0), is not supported. Please use SceneManager.LoadScene()/EditorSceneManager.OpenScene() to switch to another scene.
UnityEngine.SceneManagement.SceneManager:UnloadSceneAsync(Scene)
Unity.Scenes.GameObjectSceneRefCount:ReleaseScene(GameObjectSceneRefCount*)
Unity.Entities.ManagedComponentStore:smile:ispose()
Unity.Entities.EntityDataAccess:smile:ispose()
Unity.Entities.EntityManager:smile:estroyInstance()
Unity.Entities.WorldUnmanaged:smile:ispose()
Unity.Entities.World:smile:ispose()
Unity.Entities.World:smile:isposeAllWorlds()
Unity.Entities.DefaultWorldInitialization:smile:omainUnloadOrPlayModeChangeShutdown()

Umm so… the 2020.3.36f1 looks like it work…
2021.3.4f1 has this annoying issue.
It would be better in term of assistance if you can report the bug via editor so we can track it and having QA tracking down and bisect a little bit in between which version things start occurring.

Reported bug. CASE IN-7885

1 Like

So, I investigated a little bit.
The problem is just one in the end: even though we are building with the “Headless” option the player is still trying to create a graphics device.
You can’t run the server app from command line that use graphics acceleration like this.

There are two way to fix this problem and I will present here both.

FIRST WAY: RUN USING XVFB
You need to install on your server xfvb and either:

  • Add a `Xvfb :0 -screen 0 640x480x24 +extension GLX (or similar) your .zshrc, .bashrc or in your init.d or whatever and then execute the app with the command DISPLAY=:0 PathToMyApp/MyApp

  • Or, way simpler, use xvfb-run -a PathToMyApp/MyApp

All this may not work if you don’t have the right driver installed. The linux player support both Vulkan and OpenGL. But for GL to work, the minimum requirement is that there is OpenGL core profile 4+ available on the machine.
That may be not the case, i.e: you don’t have the latest lib mesa installed (or any other GL driver).
By installing a newest driver that support the required OpenGL core (i.e: by using the one from the ppa oibaf-graphics_drivers repo). the app should work without problems.

In case you wanna opt in for Vulknan, there is problem: the reported device when you run from the terminal is of type 4 (I think it is CPU or no graphics, don’t remember know), that looks like it is not correctly supported or is just ignored. End result, no usable devices are found and the app trigger an exception.

For sake of reference, I tried doing the following on my started from scratch VM (IIRC was ubuntu 20.0.4 but its can work also for Debian and centos in a similar way):

sudo apt install xvfb
sudo apt install mesa-utils
sudo add-apt-repository ppa:hushed:ibaf/graphics-drivers
sudo update
sudo upgrade

If you now reboot and then run the following commands:
xvfb-run -a glxinfo you should get something like:


and the server should run properly.

SECOND WAY (SUGGESTED)
Another way to fix the issue is to add the -nographics -batchmode to command line when you run your app, i.e:

Path/To/MyApp -nographics -batchmode

The server application is initialised with a null device. The sharer will fail to compile etc etc but the game run. I tried my self with the Asteroid samples, on both Linux and OSX.

A little issue that I found: if you add the -nographics but not the -batchmode the linux app crash. The osx app instead report that -batchmode is required. An inconsistent behaviour.
Also I would say -batchmode should not even be required, but this is how it work.

All that being said, the problem is the build. It should not do that, and it should work like the Unity 2020.X version.

2 Likes

To reiterate, in that specific case it is not a leak or an out of memory. At least for mono build, that is the first I tried to build.
I suppose IL2CPP is having the same issue.

I created the empty project and just build Linux server runtime then run it. It’s still failed to run properly. I expect it should work just like Unity 2020.X out of the box without needing to do any special configuration. Which Unity 2021.X version will fix this issue to make the Linux runtime works exactly like Unity 2020.X?