Huge breaking culture/language behavior change from 5.6 to 2018+

Unity Engine
1

We have a unity game that has been running on 5.6.4p1 for years and have recently updated to 2018 and found that there is a significant change to the way various functions default to culturally aware or not.

The first related issue we ran into was with periods being replaced by commas when parsing floats from string data in various places. Apparently 5.6x unity just under the hood defaulted to not require culturally invariant parameters to these sort of functions, so we got away for years with not providing those functions.

Fast forward to 2018 and now users running different OS languages and such have busted games because that behavior has changed. I’m sure it’s arguable that 5.6 was the broken behavior in this sort of situation, but someone had to have known that even so, it represented a huge underlying change in behavior that results in a lot of difficult to uncover bugs for any apps ported forward.

We thought this issue was limited to float parsing, and it wasn’t until much later that it appears that string.ToLower() (without the cultural parameter) is different in 2018 than 5.6 as well. For example, the lower case ‘i’ in a string (U+0069 : LATIN SMALL LETTER I), on some OS languages is being replaced by U+0131 : LATIN SMALL LETTER DOTLESS I, which in our case is then breaking a switch statement. No telling how many other cultural issues are hiding in our code.

One approach is obviously to go through and try and ensure that all calls to these functions use the one that takes the cultural info parameter, but this is too reactionary and impossible to be sure you have covered all the use cases of.

Has anyone else ran into this issue and found another workaround? Especially once you throw in closed source assembly plugins, unless there is a way to globally force the unity app to old behavior, one can’t be sure there aren’t still busted stuff lingering in the code to be found.

I believe we even tried to force the culture info of the app globally and iirc it didn’t seem to work in unity.

CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture;
CultureInfo.DefaultThreadCurrentUICulture = CultureInfo.InvariantCulture;
Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture

2 Likes
2

According to the following forum threads, it sounds like this issue should be fixed in 2018.3 with Scripting Runtime Version set to .NET 4.x. What Unity version are you on? Are you able to reproduce this issue with the latest 2018.3 release?

https://discussions.unity.com/t/646991/30

https://discussions.unity.com/t/544235/22

3

We’re on 2018.3.12f1

It’s not serialized data that is messed up, it’s that functions like float.TryParse, string.ToLower, etc behave differently in 2018 versus 5.6. There are tons of functions like these that have cultural overloads that are probably susceptible to this issue. Apparently in 5.6 we never had to worry about passing explicit cultural invariant parameters to these functions, because they behaved consistently on different users OS language settings, but now they don’t

4

@bobdonovan @JoshPeterson

Apologies for tagging you guys but you were both involved in other threads related to this and I was hoping you could chime in about whether this is a unity bug or whether it’s just changed behavior that the dev is meant to change their code to fix.

5

I am hoping to hear from Unity devs whether this was an intentional breaking change to the underlying libraries in Unity or a bug that should be handled by a Unity update.

6

It sounds like these are changes that came with the update to the newer version of Mono, which is the underlying c# compiler/interpreter that Unity uses. This upgraded the compiler from .Net 3.5 compatibility to .Net 4.6 compatibility, which included a new version of c#, c#6. .Net 4.6 was added in Unity 2017.1, 3.5 became deprecated in 2018.3, and was removed entirely in 2019.1.

7

That seems likely, but I would like to know if it is intentional on the part of Unity that this sort of significant behavior change occurs as a result of that upgrade, versus for example unity under the hood forcing a cultural setting as the default or whatever it was doing before.

As @bobdonovan said in one of the threads mentioned above

“Thanks for the feedback everybody. We are considering your valuable input to work out the best and safest solution for 18.3. At this point, having consistency with previous versions’ behaviours and making sure that our serialization is fully robust with InvariantCulture everywhere is more important to us than having better locale support.”

Now obviously that thread is about serialization, but I think there is an equally strong argument to be made about the consistency of the underlying library functions when it comes to basic string operations and such consistently working across Unity versions. If they decided to knowingly break consistency then so be it, I had just hoped to hear from them, so I’m not left guessing whether it is an oversight/bug or a deliberate set of changes. I have been unable to find reference to this behavioral change in change notes in more recent unity versions, which makes me think it could very well have been an oversight and potentially unintentional.

For the most part we’ve done a pass to change our code to use culturally aware versions of the relevant functions, but it is a subtle set of issues that will bite any Unity developer that ends up having players running Turkish and certain other OS languages, so whether Unity deliberately or non deliberately broke the safety of prior Unity versions has big implications for all developers.

8

Unity doesn’t write the underlying library though, that’s Mono/C#/.Net.

9

Most likely this was an unintentional change, and wasn’t caught because testing things like TryParse under various locales wasn’t a part of their QA regression suites.

As someone who works in QA primarily, I can say this would have been a hard bug to catch as it is fairly obscure, was probably unexpected to the QA team, and wouldn’t show up when testing with many of the locales where Unity teams are based. QA won’t have the resources to test their entire regression suite in every locale. They will have the majority of their tests that run in a single locale, and a subset of tests they run on various locales. This is an easy one to overlook.

10

I think I’ve tracked it down to possibly to this fix?

https://issuetracker.unity3d.com/is…66.515640864.1560468572-1976000747.1499913666

I’m having to change all the ToUpper to ToUpperInvariant and so on.

FYI this also effects your DateTime Format for Thai. I recommend changing your device locale and testing it out if you have any kind of string formatting and time.