Get symbol maps to work

claytoncurmi · September 30, 2020, 1:31pm

Hi all,

I am currently working on a solution whereby I would like to get a meaningful stack trace from a release WebGL build (currently using Unity 2020.1.4). The idea is that I can then post this on a server to be evaluated later on. I created a small demo project with the following code. I tried building 2 builds; build 1.0.5 - development build, build 1.0.6 - non-development build with debug symbols enabled. Attached are 2 screenshots for when I generate a null reference exception.

I don’t want to use FullWithStacktrace (I am using FullWithoutStacktrace) as this will be used for a live environment and such builds have poor performance. So I would like to use the stacktrace from WASM.

My questions are the following;

a…Why isn’t there a function name that includes ‘GenerateNullReferenceException’ (even though mangled it could be helpful). I noticed that when I generate a log error there is an entry with a function name that includes ‘GenerateLogError’ in it however this isn’t the case for null reference exceptions.

b…How can one make use of the debug symbols file? All entries in the 2nd screenshot are from (anonymous) methods. It gets generated and I can see it in build folder but it is never downloaded to the client (should it?). Does the symbols map file get loaded automatically?

Null reference exception stack in build 1.0.5 (development build)

Null reference exception stack in build 1.0.6 (release build)

using System;
using System.Collections;
using System.Collections.Generic;

using UnityEngine;

public class Demo : MonoBehaviour {
    // Update is called once per frame
    void Update () {
        if (Input.GetKeyDown (KeyCode.N)) {
            GenerateNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.E)) {
            GenerateLogError ();
        }

        if (Input.GetKeyDown (KeyCode.T)) {
            ThrowException ();
        }
    }

    private void GenerateNullReferenceException () {
        string s = null;
        s.Trim ();
    }

    private void GenerateLogError () {
        Debug.LogError ("Computer says no!");
    }

    private void ThrowException () {
        throw new System.Exception ("Kaboom!");
    }
}

claytoncurmi · October 2, 2020, 9:19am

Ok so today I made good improvements, however there are still some pending issues to be addressed.

I made a series of builds to identify if I can get the null reference exception stacktrace from WASM in release build. The build that got me closest to my goal was the one built with the following settings;

Which generates the following stack trace when I press the G (to execute the
GenerateGameObjectNullReferenceException method shown below).

Exception at:
Error
    at jsStackTrace (blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:22520)
    at stackTrace (blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:22691)
    at blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:9961
    at <anonymous>:wasm-function[32174]:0x89430f
    at <anonymous>:wasm-function[32028]:0x88fd33
    at <anonymous>:wasm-function[9307]:0x432a07
    at <anonymous>:wasm-function[10278]:0x47e42c
    at <anonymous>:wasm-function[24191]:0x70a0d5
    at <anonymous>:wasm-function[24189]:0x70a03a
    at <anonymous>:wasm-function[31650]:0x881418
    at <anonymous>:wasm-function[31646]:0x881144
    at <anonymous>:wasm-function[14818]:0x5a196b
    at <anonymous>:wasm-function[31778]:0x886504
    at <anonymous>:wasm-function[32043]:0x88fe25
    at <anonymous>:wasm-function[2951]:0x14b83a
    at <anonymous>:wasm-function[2950]:0x14b7a3
    at <anonymous>:wasm-function[7339]:0x316059
    at <anonymous>:wasm-function[7327]:0x314ba3
    at <anonymous>:wasm-function[9635]:0x448a31
    at <anonymous>:wasm-function[9634]:0x44875a
    at <anonymous>:wasm-function[8080]:0x367d4e
    at <anonymous>:wasm-function[7431]:0x31f356
    at <anonymous>:wasm-function[7431]:0x31f36d
    at <anonymous>:wasm-function[7426]:0x31e583
    at <anonymous>:wasm-function[7420]:0x31cdfa
    at dynCall_v (<anonymous>:wasm-function[32306]:0x89919e)
    at Object.dynCall_v (blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:469584)
    at browserIterationFunc (blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:131380)
    at Object.runIter (blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:134453)
    at Browser_mainLoop_runner (blob:http://localhost:50247/856151ad-578d-4a1a-9d24-39e43f50d98f:2:132915)

This stacktrace is good enough for me as it can be translated to the following using the symbols map;

__ZN6il2cpp2vm9Exception5RaiseEP15Il2CppExceptionP10MethodInfo|il2cpp::vm::Exception::Raise(Il2CppException*, MethodInfo*)
_il2cpp_raise_exception
__Z25scripting_raise_exception21ScriptingExceptionPtr|scripting_raise_exception(ScriptingExceptionPtr)
__ZN9Scripting24RaiseNullExceptionObjectE18ScriptingObjectPtr|Scripting::RaiseNullExceptionObject(ScriptingObjectPtr)
__Z21Object_CUSTOM_GetNameP12Il2CppObject|Object_CUSTOM_GetName(Il2CppObject*)
_Object_GetName_m6F0498EEECA37CD27B052F53ECDDA019129F3D7A
_Object_get_name_m0C7BC870ED2F0DC5A2FB09628136CD7D1CB82CFB
_Demo_GenerateGameObjectNullReferenceException_mC23228EE215A36BBFC1483C71036D4D590921B9E
_Demo_Update_mA20CD8F8E6C14018B5051CFC4624F1559E45BA40
__Z65RuntimeInvoker_TrueVoid_t700C6383A2A510C2CF4DD86DABD5CA9FF70ADAC5PFvvEPK10MethodInfoPvPS4_|RuntimeInvoker_TrueVoid_t700C6383A2A510C2CF4DD86DABD5CA9FF70ADAC5(void (*)(), MethodInfo const*, void*, void**)
__ZN6il2cpp2vm7Runtime6InvokeEPK10MethodInfoPvPS5_PP15Il2CppException|il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**)
_il2cpp_runtime_invoke
__Z23scripting_method_invoke18ScriptingMethodPtr18ScriptingObjectPtrR18ScriptingArgumentsP21ScriptingExceptionPtrb|scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)
__ZN19ScriptingInvocation6InvokeEP21ScriptingExceptionPtrb|ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)
__ZN13MonoBehaviour16CallUpdateMethodEi|MonoBehaviour::CallUpdateMethod(int)
__ZN13MonoBehaviour6UpdateEv|MonoBehaviour::Update()
__ZN20BaseBehaviourManager12CommonUpdateI16BehaviourManagerEEvv|void BaseBehaviourManager::CommonUpdate()
__ZN13UpdateManager6UpdateEv|UpdateManager::Update()
__ZZ23InitPlayerLoopCallbacksvEN41UpdateScriptRunBehaviourUpdateRegistrator7ForwardEv|InitPlayerLoopCallbacks()::UpdateScriptRunBehaviourUpdateRegistrator::Forward()
__Z17ExecutePlayerLoopP22NativePlayerLoopSystem|ExecutePlayerLoop(NativePlayerLoopSystem*)
__Z17ExecutePlayerLoopP22NativePlayerLoopSystem|ExecutePlayerLoop(NativePlayerLoopSystem*)
__Z10PlayerLoopv|PlayerLoop()
__ZL8MainLoopv|MainLoop()
dynCall_v

However when I try to execute the method
GenerateStringNullReferenceException or
GeneratePlayerNullReferenceException no exception is raised and it proceeds as if they were properly initialized. At this point I expect that both will fail on line 58 and line 75 respectively. Why does it crash in GenerateGameObjectNullReferenceException but not the others?

using System;
using System.Collections;
using System.Collections.Generic;

using UnityEngine;

public class Player {
    private int health;

    public int Health {
        get {
            return health;
        }
    }

    public bool IsDead {
        get {
            return health == 0;
        }
    }
    public void ApplyDamage (int damage) {
        health = Mathf.Max (0, health - damage);
    }

}

public class Demo : MonoBehaviour {
    // Update is called once per frame
    void Update () {
        if (Input.GetKeyDown (KeyCode.S)) {
            GenerateStringNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.H)) {
            HandleStringNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.P)) {
            GeneratePlayerNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.G)) {
            GenerateGameObjectNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.E)) {
            GenerateLogError ();
        }

        if (Input.GetKeyDown (KeyCode.T)) {
            ThrowException ();
        }
    }

    private void GenerateStringNullReferenceException () {
        Debug.Log ("Generating null reference exception");
        string s = null;
        s.Trim ();
        Debug.Log (s.Length);
    }

    private void HandleStringNullReferenceException () {
        Debug.Log ("Handling null reference exception");
        try {
            string s = null;
            s.Trim ();
            Debug.Log (s.Length);
        } catch (Exception ex) {
            Debug.LogError (ex.Message);
        }
    }

    private void GeneratePlayerNullReferenceException () {
        Player p = null;
        p.ApplyDamage (5);
        Debug.Log ("Health - " + p.Health);
        Debug.Log ("Is dead - " + p.IsDead);
    }

    private void GenerateGameObjectNullReferenceException () {
        var go = GameObject.Find ("NonExistentObject");
        Debug.Log (go.name);
    }

    private void GenerateLogError () {
        Debug.LogError ("Computer says no!");
    }

    private void ThrowException () {
        throw new System.Exception ("Kaboom!");
    }
}

claytoncurmi · October 5, 2020, 8:38am

Another update regarding the code below. Why does the method GenerateStringNullReferenceExceptionOriginal does not generate a null reference exception but GenerateStringNullReferenceExceptionContains does?

using System;
using System.Collections;
using System.Collections.Generic;

using UnityEngine;

public class Player {
    private int health;

    public int Health {
        get {
            return health;
        }
    }

    public bool IsDead {
        get {
            return health == 0;
        }
    }
    public void ApplyDamage (int damage) {
        health = Mathf.Max (0, health - damage);
    }

    public void PrintName () {
        Debug.Log ("Player One");
    }

}

public class Demo : MonoBehaviour {
    // Update is called once per frame
    void Update () {
        if (Input.GetKeyDown (KeyCode.Q)) {
            GenerateStringNullReferenceExceptionOriginal ();
        }

        if (Input.GetKeyDown (KeyCode.W)) {
            GenerateStringNullReferenceExceptionTrim ();
        }

        if (Input.GetKeyDown (KeyCode.E)) {
            GenerateStringNullReferenceExceptionContains ();
        }

        if (Input.GetKeyDown (KeyCode.R)) {
            HandleStringNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.T)) {
            GeneratePlayerNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.Y)) {
            GenerateGameObjectNullReferenceException ();
        }

        if (Input.GetKeyDown (KeyCode.U)) {
            GenerateLogError ();
        }

        if (Input.GetKeyDown (KeyCode.I)) {
            ThrowException ();
        }
    }

    private void GenerateStringNullReferenceExceptionOriginal () {
        Debug.Log ("Generating null reference exception (original)");

        string s = null;
        s.Trim ();
        Debug.Log (s.Length);
    }

    private void GenerateStringNullReferenceExceptionContains () {
        Debug.Log ("Generating null reference exception (contains)");
        string s = null;

        if (s == null) {
            Debug.Log ("String is null!");

            s.Contains ("Hello");

            if (s != null) {
                Debug.Log ("String is not null!");
            }

            Debug.Log (s.Length);
        } else {
            s.Trim ();
            Debug.Log (s.Length);
        }
    }

    private void GenerateStringNullReferenceExceptionTrim () {
        Debug.Log ("Generating null reference exception (trim)");
        string s = null;

        if (s == null) {
            Debug.Log ("String is null!");

            s.Trim ();

            if (s != null) {
                Debug.Log ("String is not null!");
            }

            Debug.Log (s.Length);
        } else {
            s.Trim ();
            Debug.Log (s.Length);
        }
    }

    private void HandleStringNullReferenceException () {
        Debug.Log ("Handling null reference exception");
        try {
            string s = null;
            s.Trim ();
            Debug.Log (s.Length);
        } catch (Exception ex) {
            Debug.LogError (ex.Message);
        }
    }

    private void GeneratePlayerNullReferenceException () {
        Player p = null;

        if (p == null) {
            Debug.Log ("Player is null!");

            p.PrintName ();

            if (p != null) {
                Debug.Log ("Player is not null!");
            }

            Debug.Log (p.Health);
        } else {
            p.ApplyDamage (5);
            Debug.Log ("Health - " + p.Health);
            Debug.Log ("Is dead - " + p.IsDead);
        }
    }

    private void GenerateGameObjectNullReferenceException () {
        var go = GameObject.Find ("NonExistentObject");
        Debug.Log (go.name);
    }

    private void GenerateLogError () {
        Debug.LogError ("Computer says no!");
    }

    private void ThrowException () {
        throw new System.Exception ("Kaboom!");
    }
}

jukka_j · October 5, 2020, 6:47pm

In order to have the build retain the function callstack names, instead of showing those wasm-function[1234], you can pass the directive

PlayerSettings.WebGL.emscriptenArgs = " --profiling-funcs ";

to a build in an Editor script. That will include the function symbol names in the .wasm file. Do notice however that including symbols does increase the size of the .wasm file by some amount.

You can try using the attached buildWebGL.cs build script. (check its contents for documentation)

6412182–716235–buildWebGL.cs (6.92 KB)

jukka_j · October 5, 2020, 6:48pm

Oh, forgot to mention - the option Enable Exceptions: None/Explicit/FullWithStacktrace does not relate to the presence of the function symbol names, but rather to the presence of try-catch clauses in the generated code.

claytoncurmi · October 6, 2020, 7:25am

Thanks for your reply.

I have recovered the required function names using the debugSymbols options (set to true) as explained in post #2 - https://discussions.unity.com/t/810790/2 . This generated a JSON file which includes the ID and associated function name. Using a simple lookup code I was able to convert wasm-function[1234] to the required function name.

The problem I have is as explained at the end of the same post, that certain functions trigger a null exception (for example GenerateGameObjectNullReferenceException) but not others such as GenerateStringNullReferenceException when they should. In fact the same code when executed in development build will generate the null reference (which is correct). I also posted another update with my latest findings as you can see above in post #3.

Could you indicate why this happens and how can we avoid this from happening?

claytoncurmi · October 6, 2020, 9:05am

In a nutshell these are the results I am getting with these two builds (using the code from post #3);

Build A

Enable Exceptions - FullWithoutStacktrace
Build Type - Release
Debug Symbols - Enabled
Stack Trace - Full (for all log types)

Press ‘Q’ - Generates exception but no debug symbols in stack trace
Press ‘W’ - Generates exception but no debug symbols in stack trace
Press ‘E’ - Generates exception but no debug symbols in stack trace
Press ‘R’ - Handles exception and outputs error log with debug symbols in stack trace
Press ‘T’ - Generates exception but no debug symbols in stack trace
Press ‘Y’ - Generates exception but no debug symbols in stack trace
Press ‘U’ - Outputs error log with debug symbols in stack trace
Press ‘I’ - Generates exception but no debug symbols in stack trace

Build B

Enable Exceptions - None
Build Type - Release
Debug Symbols - Enabled
Stack Trace - Full (for all log types)

Press ‘Q’ - No error
Press ‘W’ - No error
Press ‘E’ - Generates exception and has debug symbols in stack trace
Press ‘R’ - No error
Press ‘T’ - No error
Press ‘Y’ - Generates exception and has debug symbols in stack trace
Press ‘U’ - Outputs error log with debug symbols in stack trace
Press ‘I’ - Generates exception and has debug symbols in stack trace

As you can see these are totally different outputs. What I want is to have the same exceptions fired as in Build A with debug symbols in the stack trace. Is that possible?

jukka_j · October 6, 2020, 9:21am

Hmm, yeah, try doing a Release build with Full with stack trace, Debug Symbols - Enabled, Stack Trace - Full, but do the build via the buildWebGL.cs script I posted.

In the build dropdown that appears, try out the option “HTML5 Export/Wasm+Release+Profiling uncompressed…”. Does that give the stack trace information?

claytoncurmi · October 6, 2020, 10:34am

So I imported the script attached in the previous post and made two builds as follows;

Build A

Enable Exceptions - FullWithoutStackTrace
Ran the command “HTML5 Export/Wasm+Release+Profiling uncompressed…”

This generates a stack trace that is identical to the output as in my first post. As indicated again below there is no mention of the method that triggered the exception and therefore I would be unable to track the issue down.

Build B

Enable Exceptions - FullWithStackTrace
Ran the command “HTML5 Export/Wasm+Release+Profiling uncompressed…”

This generates a partial stack trace in the message (printed in red) as shown below. But as pointed in the documentation “FullWithStackTrace exception support will decrease performance and increase browser memory usage. Only use this for debugging purposes and make sure to test in a 64-bit browser”. As this is a tool meant to run in a release environment, this option is not possible.

As indicated in post #7, the best output I had so far in terms of logging is when I made a build with Enable Exceptions set to None. In this case when I press E, Y or I, a log as shown below is outputted (the one on the left is the actual log given from the WebGL player, the one on the right is how the stacktrace after using the lookup symbols map) and as you can see on line 8 it indicates the origin of the problematic code which is great. The problem is that the code doesn’t always crash when it should as when pressing Q, W, R or T.

jukka_j · October 6, 2020, 11:40am

This seems like the method may have gotten inlined, or some other issue. This may be worth filing a bug.

I believe this is something we have fixed just recently, in a feature that adds such safety checks even when Enable Exceptions is set to None.

claytoncurmi · October 6, 2020, 12:42pm

Ok thanks for you reply,

So when calling s.Trim() on the null string variable ‘s’ or accessing the method of a non-initialised instance of the Player class, the WebGL player should crash, irrespective of Enable Exceptions value, right?

Is this the bug fix you are referring to Unity Issue Tracker - [IL2CPP] Exceptions in looped finally blocks are not thrown in IL2CPP builds? In which version will it be available? Thanks once again.

De-Panther · October 6, 2020, 12:45pm

claytoncurmi · October 6, 2020, 12:49pm

I am asking if that is the actual bug fix as I am not sure. If it is, it didn’t solve my issue as I tried 2020.1.7f1 already and same results as before.

jukka_j · October 6, 2020, 4:47pm

Currently if null.Function() is called, the WebGL build will:

happily truck through the line to a random crash or no-crash if the page was built with Player->Publishing Settings->Enable Exceptions set to Explicit or None, and
throw an exception if the page was built with Enable Exceptions set to Full (with or without Stacktrace)

The first mode is good for generating smallest code size and performance for the web in the Release scenario where one has already debugged through all kinds of code issues like null references. The second mode is good for debugging, but comes with quite a large penalty on WebAssembly code size and startup load times, and to a smaller degree to runtime performance.

Recently we realized that we are missing a third kind of mode in between, where we would safely abort execution (by raising a WebAssembly trap) if null.Function() is met, but without the penalty on code size, load times or performance. That mode would be good for shipping final projects, while still retaining the ability to get meaningful error reports back to analytics (by e.g. JavaScript window.onerror() based error handlers)

This mode is not listed under the issue tracker since it was observed based on our internal communication, and not due to an error report that was submitted.

Further discussing with our codegen team today, it does look like the function did get inlined. That brings a tricky situation: I understood you would like to get these reports from Release builds. Because inlining is critical for small code size, disabling inlining to get the stack trace would cause quite a large code size penalty that might not be worth the tradeoff. Building in debug mode would limit the amount of inlining, but that would definitely run quite slow. Not sure what to suggest here…

claytoncurmi · October 7, 2020, 8:36am

Ok thanks for your detailed explanation, much appreciated.

Unfortunately this won’t allow us to track the origin of the issue and might cause unwanted and misleading side effects. Is there a different approach that we can take to get meaningful error logs from a release WebGL environment (except using the FullWithStacktrace option) with the currently available Unity versions?

In conclusion, what we are truly really after is the 3rd mode whereby an exception is fired for null.Function() without penalty on code size or performance. Do you have an indication when this feature will be available and in which version?

jukka_j · October 7, 2020, 8:49am

Inlining may be something that happens aggressively in small synthetic test cases like shown above, but in real world code, it may be less common, depending on the sizes of the functions and how many times they are called. Unfortunately I do not have a solution here. Do note that the same inlining will occur on all IL2CPP platforms, not just specifically on WebGL.
[/QUOTE]

Sorry, we do not have visibility to that yet.

claytoncurmi · October 7, 2020, 8:55am

Ok thanks for your time.

RMGK · September 1, 2021, 12:56am

This worked like a charm! Thank you!

Topic		Replies	Views
How to use WebGL Debug Symbols Unity Engine Performance , Platforms	1	1774	December 1, 2020
How to get good stack trace on WebGL Unity Engine Platforms	8	3971	November 17, 2018
Understanding native crashes and symbols with StackTrace Utility Unity Engine Android , Platforms , Question	4	4600	September 22, 2020
WebGL NullReferenceException with no stacktrace in Performance Reports Unity Engine Platforms	3	2098	April 16, 2018
[...or UI?] NullReferenceException with no stack trace, with Stack Trace Logging>Full Unity Engine Animation , 5-3-beta	3	6406	December 4, 2015

Get symbol maps to work

Build A

Enable Exceptions - FullWithoutStacktrace Build Type - Release Debug Symbols - Enabled Stack Trace - Full (for all log types)

Build B

Enable Exceptions - None Build Type - Release Debug Symbols - Enabled Stack Trace - Full (for all log types)

Build A

Build B

Related topics

Enable Exceptions - FullWithoutStacktrace
Build Type - Release
Debug Symbols - Enabled
Stack Trace - Full (for all log types)

Enable Exceptions - None
Build Type - Release
Debug Symbols - Enabled
Stack Trace - Full (for all log types)