General feedback on localization

[TimHeijden2]

I think it’s a great idea for Unity to provide an API like this, It’s something that so many projects use nowadays that having a robust system in place will save time & maybe improve quality of implementation as well!

As many other probably have, we implemented our own localization system and I’d like to share some experiences we’ve had as feedback.

Unfortunately, I wasn’t able to test the preview because I couldn’t create an asset or string table because of an exception throwing when opening the window.:
ArgumentOutOfRangeException Index was out of range UnityEditor.Localization.AssetTableCollection.get_TableName () (at Library/PackageCache/com.unity.localization@0.2.1-preview/Editor/LocalizationPlayerSettings.cs:116)

It may have been because I did something out of order, but I don’t currently have time to retry the process. I might make another post when I do.

1. String localization on text objects
In our localization system, we also have a component on the text object that needs to be localized. This component changes the text value on Awake/Start/OnEnable, an unfortunate side effect of this is that sometimes the original/old text is visible for 1 frame and then change. This is not only visually weird but also requires text to be rebuilt 2x instead of 1x. In one of our mobile games, this causes some performance issues due to a custom script making text more expensive to rebuild. (custom outline & gradient)

Something we’ve been considering is to create a custom override for the Text (& TMPro text) component and write the localization inside that component to solve this issue, but we haven’t gotten around to it.

I’d love to see you guys taking a look at this (probably common?) problem as well.

2. Memory / Resources / AssetBundles
An important factor that I’d like to bring up is memory management. I find it very important that we be able to load ONLY the data of a specific language instead of having string/assets of other languages in memory. It looks like you guys have already taken care of this though! (reading documentation)

3. Spreadsheet support
Naturally, also mentioned by someone else in another thread. It is crucial we are able to import data from a spreadsheet. And I’d like to reiterate that we should be able to do this from code, such that we can write scripts to download & import the localization as & when we please.

Please make sure this works outside play mode as well, so we can import & process localization beforehand.

4. Font change based on language
Another scenario we’ve solved in our custom UI system is to change the font used on a text component based on language. This is because many fonts don’t support more complex languages (chinese traditional, japanese, korean, arabic etc.) and we need to work with alternatives.

In a specific game, we have 2 fonts in use for latin languages. One for “titles”, and one for all the other stuff. When the localization would be set, the component that handles localization not only changes the text, but also the font if needed.

One nice addition to this we used was the ability to retrieve a “most likely” installed font on the system OS, to prevent having to put the font in the game and increase file size. This was done through Font.GetOSInstalledFontNames() & Font.CreateDynamicFontFromOSFont().

Also a minor note to keep in mind: Whereas Standalone builds have a “fallback” font in place to use if an unknown character of a font is used, not each platform supports this.

While I wouldn’t say having a system like this in place inside this API, if it won’t be added I’d like to make sure we are able to hook into it and apply this font-swapping based on the selected language. (Without needing to rebuild a text extra time! see point #1 ^^)

5. Multi-platform support
Another important point I’d like to make is about this system being multi-platform. We had to write a few bits of custom code to make localization work for some platforms. In one case, this also included doing some programming in native code to make retrieving the locale work. In addition, it seems that some platforms use different enumerations for language and as such I find it risky to depend on something like “Application.systemLanguage” as long as that the issues mentioned above aren’t resolved. CultureInfo also wasn’t working for those platforms.

Preferably, instead of requiring every developer to make a custom locale provider, I suggest making the language work for all platforms such that making the providers is only necessary if a developer wants to actually create a different way of determining locale.

6. “Testability” / workflow
Finally, I’d like to point out general testability/workflow struggles we’ve had in the past in terms of localization linkage. During development & QA, we’d get stuff back like:

• A text component not having the localization component (forgot/removed by devs)
• Incorrectly matched localization keys (invalid due to import, typo, etc.)
• Localization that doesn’t fit the text component

I actually haven’t “solved” this issue yet (again due to not getting around to it) but I can imagine writing various editor scripts to find such possible issues on build or through a visual toggle showing each text that is localized. No concrete ideas on that, but perhaps you guys do!

That’s pretty much it, love to hear your thoughts.

[karl_jones]

Thanks for the feedback!

@TimHeijden2

Unfortunately, I wasn’t able to test the preview because I couldn’t create an asset or string table because of an exception throwing when opening the window.:
ArgumentOutOfRangeException Index was out of range UnityEditor.Localization.AssetTableCollection.get_TableName () (at Library/PackageCache/com.unity.localization@0.2.1-preview/Editor/LocalizationPlayerSettings.cs:116)

This is a known bug. We will have a fix in the next release.

@TimHeijden2

1. String localization on text objects
In our localization system, we also have a component on the text object that needs to be localized. This component changes the text value on Awake/Start/OnEnable, an unfortunate side effect of this is that sometimes the original/old text is visible for 1 frame and then change. This is not only visually weird but also requires text to be rebuilt 2x instead of 1x. In one of our mobile games, this causes some performance issues due to a custom script making text more expensive to rebuild. (custom outline & gradient)

Something we’ve been considering is to create a custom override for the Text (& TMPro text) component and write the localization inside that component to solve this issue, but we haven’t gotten around to it.

I’d love to see you guys taking a look at this (probably common?) problem as well.

This can happen if the localization data is not available(not loaded yet), we have asynchronous operations that can be yielded on while the loading happens. You can also set it to preload all the localization data so that it’s immediately available, just yield on the preload operation in a loading screen. In the future, we will be looking at deeper integration with Unity components such as TMP so hopefully, we can provide a seamless solution without the issues you describe.

@TimHeijden2

2. Memory / Resources / AssetBundles
An important factor that I’d like to bring up is memory management. I find it very important that we are able to load ONLY the data of a specific language instead of having string/assets of other languages in memory. It looks like you guys have already taken care of this though! (reading documentation)

Yes, this was something we considered from the start. We want the localization to be usable by both small games that only require a handful of text and large RPGS with hundreds of thousands of assets and strings. We have the ability to preload all data for a locale or only load on demand. Assets can be unloaded when they are no longer required. We are built on top of the Addressables system which is built for this purpose so all the hard work is already done

@TimHeijden2

3. Spreadsheet support
Naturally, also mentioned by someone else in another thread. It is crucial we are able to import data from a spreadsheet. And I’d like to reiterate that we should be able to do this from code, such that we can write scripts to download & import the localization as & when we please.

Please make sure this works outside play mode as well, so we can import & process localization beforehand.

Yes, this is planned. Good point about the scripting api. We are recording everything people suggest so its still good to reiterate stuff people have said, shows how important it is

@TimHeijden2

4. Font change based on language
Another scenario we’ve solved in our custom UI system is to change the font used on a text component based on language. This is because many fonts don’t support more complex languages (chinese traditional, japanese, korean, arabic etc.) and we need to work with alternatives.

In a specific game, we have 2 fonts in use for latin languages. One for “titles”, and one for all the other stuff. When the localization would be set, the component that handles localization not only changes the text, but also the font if needed.

One nice addition to this we used was the ability to retrieve a “most likely” installed font on the system OS, to prevent having to put the font in the game and increase file size. This was done through Font.GetOSInstalledFontNames() & Font.CreateDynamicFontFromOSFont().

Also a minor note to keep in mind: Whereas Standalone builds have a “fallback” font in place to use if an unknown character of a font is used, not each platform supports this.

While I wouldn’t say having a system like this in place inside this API, if it won’t be added I’d like to make sure we are able to hook into it and apply this font-swapping based on the selected language. (Without needing to rebuild a text extra time! see point #1 ^^)

A lot of the font based stuff will be done through TMP although we will look at areas we can hook in. One way to support fonts at the moment is with CustomLocales, you can assign a font per locale.

@TimHeijden2

5. Multi-platform support
Another important point I’d like to make is about this system being multi-platform. We had to write a few bits of custom code to make localization work for some platforms. In one case, this also included doing some programming in native code to make retrieving the locale work. In addition, it seems that some platforms use different enumerations for language and as such I find it risky to depend on something like “Application.systemLanguage” as long as that the issues mentioned above aren’t resolved. CultureInfo also wasn’t working for those platforms.

Preferably, instead of requiring every developer to make a custom locale provider, I suggest making the language work for all platforms such that making the providers is only necessary if a developer wants to actually create a different way of determining locale.

Interesting. The loading of Locales and assets is handled by the Addressables system. We do support CultureInfo, however, its not a dependency, its more optional. We are aware of some issues with getting the correct system language. If you know a specific platforms then feel free to either report a bug(if its a general Unity issue) or post it here so we can make a note and look into it.

@TimHeijden2

6. “Testability” / workflow
Finally, I’d like to point out general testability/workflow struggles we’ve had in the past in terms of localization linkage. During development & QA, we’d get stuff back like:

• A text component not having the localization component (forgot/removed by devs)
• Incorrectly matched localization keys (invalid due to import, typo, etc.)
• Localization that doesn’t fit the text component

I actually haven’t “solved” this issue yet (again due to not getting around to it) but I can imagine writing various editor scripts to find such possible issues on build or through a visual toggle showing each text that is localized. No concrete ideas on that, but perhaps you guys do!

That’s pretty much it, love to hear your thoughts.

Thanks. We will bear these in mind.

[TimHeijden2]

Thanks for the response, I’ll make sure to take a closer look to the workings once that new version comes out. (after that I can maybe expand on #1 & #4)

@karl_jones

@TimHeijden2

5. Multi-platform support
In addition, it seems that some platforms use different enumerations for language and as such I find it risky to depend on something like “Application.systemLanguage” as long as that the issues mentioned above aren’t resolved. CultureInfo also wasn’t working for those platforms.

Preferably, instead of requiring every developer to make a custom locale provider, I suggest making the language work for all platforms such that making the providers is only necessary if a developer wants to actually create a different way of determining locale.

Interesting. The loading of Locales and assets is handled by the Addressables system. We do support CultureInfo, however, its not a dependency, its more optional.

I’m not sure exactly what you mean by this. Would we use a CultureInfo string to retrieve a specific Locale asset? (through the Addressables system)

The part I quoted is really about the standard CultureInfo C# class to retrieve a string not working for all platforms. Personally, I WOULD ALWAYS prefer a string like “en-US” over an arbitrary enumeration because it would likely be incomplete (or contain a ridiculous amount of entries) so if you ever need to choose I hope the system will lean towards the string version!

[AlkisFortuneFish]

@karl_jones

In the future, we will be looking at deeper integration with Unity components such as TMP so hopefully, we can provide a seamless solution without the issues you describe.

As a note, our internal localisation system is implemented as a custom TMP tag, that pulls the appropriate string from our localisation assets. That works at a low enough level in TMP that all methods work as expected, even SetText with format arguments, plus the string is always displayed correctly for the selected locale and updated in real-time as the localisation assets change, in edit mode and play mode.

Some similar deep integration between this system and Unity’s text components would be very welcome, as feature-wise what you have developed here eclipses the system I came up with.

[karl_jones]

@AlkisFortuneFish

As a note, our internal localisation system is implemented as a custom TMP tag, that pulls the appropriate string from our localisation assets. That works at a low enough level in TMP that all methods work as expected, even SetText with format arguments, plus the string is always displayed correctly for the selected locale and updated in real-time as the localisation assets change, in edit mode and play mode.

Some similar deep integration between this system and Unity’s text components would be very welcome, as feature-wise what you have developed here eclipses the system I came up with.

Deeper integrations is certainly on the road map. For now we are trying to keep it all in the package but I’m sure that we will have to make changes to components further down.