Generating key constants per localization table

Hi!

So, I’m toying with the localization for a while now and I always (literally) end up manually updating the texts for various reasons. So I developed this (or iterations of this):
See the improved example below

using System.Collections.Generic;
using UnityEngine.Localization.Settings;
using UnityEngine.ResourceManagement.AsyncOperations;

namespace LurkingNinja.MyGame
{
    public static class I18N
    {
        public const string AYS = "AYS";
        public const string AYS_QUESTION = "MESSAGE";
        public const string TO_DESKTOP = "TO_DESKTOP";
        public const string TO_MENU = "TO_MENU";

        public const string GENERIC = "Generic";
        public const string CANCEL = "CANCEL";
        public const string OPTIONS = "OPTIONS";
        public const string QUIT = "QUIT";

        public const string MAIN_MENU = "MainMenu";
        public const string SINGLE = "SINGLE";
        public const string MULTI = "MULTI";
        public const string CREDITS = "CREDITS";

        public const string VERSION = "VERSION";

        public static string Get(in string table, in string key) => LocalizationSettings.StringDatabase
                .GetLocalizedString(table, key);

        public static string Get(in string table, in string key, IList<object> smart) => LocalizationSettings
                .StringDatabase.GetLocalizedString(table, key, smart);

        public static AsyncOperationHandle<string> GetAsync(in string table, in string key) => LocalizationSettings
                .StringDatabase.GetLocalizedStringAsync(table, key);

        public static AsyncOperationHandle<string>  GetAsync(in string table, in string key, IList<object> smart) =>
                LocalizationSettings.StringDatabase
                        .GetLocalizedStringAsync(table, key, smart);
    }
}

My question / feature request is to have an option to automatically generate a constant-list file with a static accessible class or whatever for me per table, including the table’s reference. Something like the consts in this script above only in separate generated files. More and more Unity tools do this and I like it. I’d like that Localization would join to the generate source code revolution. :slight_smile:

We do actually have this down as something to look into in the future.
The idea was to do something similar and to use the Key id with the Key as the name.
E.G

public enum MyTable : long
{
 START_GAME = 3423423423,
 EXIT_GAME. = 4534534534,
 // etc
}

It’s likely to be something we look into further once we are out of pre-release, so more of a longer-term feature. There’s some areas that need some thinking about, keeping the code in sync especially if the tables are coming from some other sources etc.
Thanks for the code snippet, ill add it to our task for future reference.
I imagine you could implement something like this as an extension now :smile:

1 Like

In the mean time I came up with this. Obviously quite crude, but it works for now. :slight_smile:
See the latest tone below.

using System;
using System.IO;
using System.Text;
using UnityEditor;
using UnityEngine;
using UnityEngine.Localization.Tables;
using Object = UnityEngine.Object;

namespace LurkingNinja.MyGame.Editor
{
    public class OnAssetPostProcess : AssetPostprocessor
    {
        private const string PATH = "/_Eclypsia/Code/_generated/i18n/";
 
        private static string GetFullPath(string fileName) => $"{Application.dataPath}{PATH}{fileName}.cs";

        private static void GenerateTableAccessorFile(SharedTableData table, string fileName)
        {
            var genPath = GetFullPath(fileName);
            using var writer = new StreamWriter(genPath, false);
            writer.WriteLine(GenerateFileContent(table));
            AssetDatabase.ImportAsset($"Assets{PATH}{fileName}.cs");
        }
 
        private static string GenerateFileContent(SharedTableData table)
        {
            var sb = new StringBuilder();
            sb.Append("//------------------------------------------------------------------------------");
            sb.Append(Environment.NewLine);
            sb.Append("// <auto-generated>");
            sb.Append(Environment.NewLine);
            sb.Append("//     This code was generated by a tool.");
            sb.Append(Environment.NewLine);
            sb.Append($"//     Generated: {DateTime.Now}");
            sb.Append(Environment.NewLine);
            sb.Append("//");
            sb.Append(Environment.NewLine);
            sb.Append("//     Changes to this file may cause incorrect behavior and will be lost if");
            sb.Append(Environment.NewLine);
            sb.Append("//     the code is regenerated.");
            sb.Append(Environment.NewLine);
            sb.Append("// </auto-generated>");
            sb.Append(Environment.NewLine);
            sb.Append("//------------------------------------------------------------------------------");
            sb.Append(Environment.NewLine);
            sb.Append(Environment.NewLine);
            sb.Append("namespace LurkingNinja.MyGame.Internationalization");
            sb.Append(Environment.NewLine);
            sb.Append("{");
            sb.Append(Environment.NewLine);
            sb.Append($"\tpublic static class i18n_{table.TableCollectionName}");
            sb.Append(Environment.NewLine);
            sb.Append("\t{");
            sb.Append(Environment.NewLine);
            sb.Append($"\t\tpublic const string NAME = \"{table.TableCollectionName}\";");     
            sb.Append(Environment.NewLine);
            foreach (var entry in table.Entries)
            {
                sb.Append($"\t\tpublic const long {entry.Key} = {entry.Id};");     
                sb.Append(Environment.NewLine);
            }
            sb.Append("\t}");
            sb.Append(Environment.NewLine);
            sb.Append("}");
            sb.Append(Environment.NewLine);
            return sb.ToString();
        }

        private static string GetFileName(string fileName) =>
                Path.GetFileNameWithoutExtension(fileName).Replace(" ", "_");

        private static void OnPostprocessAllAssets(
            string[] importedAssets, string[] deletedAssets, string[] movedAssets, string[] movedFromAssetPaths)
        {
            foreach (var path in importedAssets)
            {
                var obj = AssetDatabase.LoadAssetAtPath<Object>(path);
                if(obj is not SharedTableData tableData) continue;
                GenerateTableAccessorFile(tableData, GetFileName(path));
            }

            foreach (var path in deletedAssets)
            {
                var fileName = GetFileName(path);
                if (File.Exists(GetFullPath(fileName))) File.Delete(GetFullPath(fileName));
            }
        }
    }
}

And the generated file looks like this:

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Generated: 6/2/2021 5:20:12 PM
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

namespace LurkingNinja.MyGame.Internationalization
{
   public static class i18n_Generic
   {
      public const string NAME = "Generic";
      public const long CANCEL = 9059086336;
      public const long OPTIONS = 38761382981632;
      public const long QUIT = 38937908654080;
   }
}

The query helper

using System.Collections.Generic;
using UnityEngine.Localization.Settings;
using UnityEngine.ResourceManagement.AsyncOperations;

namespace LurkingNinja.MyGame.Internationalization
{
    public static class I18N
    {
        public static string Get(in string table, in long key) => LocalizationSettings.StringDatabase
                .GetLocalizedString(table, key);

        public static string Get(in string table, in long key, IList<object> smart) => LocalizationSettings
                .StringDatabase.GetLocalizedString(table, key, smart);

        public static AsyncOperationHandle<string> GetAsync(in string table, in long key) => LocalizationSettings
                .StringDatabase.GetLocalizedStringAsync(table, key);

        public static AsyncOperationHandle<string>  GetAsync(in string table, in long key, IList<object> smart) =>
                LocalizationSettings.StringDatabase
                        .GetLocalizedStringAsync(table, key, smart);
    }
}

Usage example (with UI Toolkit):

_buttons[MAIN_MENU].Q<Button>("Quit").text = I18N.Get(i18n_Generic.NAME, i18n_Generic.QUIT);
1 Like

See the new one below.

My generator evolved a bit:

        private static string GenerateFileContent(SharedTableData table)
        {
            var sb = new StringBuilder();
            sb.Append("//------------------------------------------------------------------------------");
            sb.Append(Environment.NewLine);
            sb.Append("// <auto-generated>");
            sb.Append(Environment.NewLine);
            sb.Append("//     This code was generated by a tool.");
            sb.Append(Environment.NewLine);
            sb.Append($"//     Generated: {DateTime.Now}");
            sb.Append(Environment.NewLine);
            sb.Append("//");
            sb.Append(Environment.NewLine);
            sb.Append("//     Changes to this file may cause incorrect behavior and will be lost if");
            sb.Append(Environment.NewLine);
            sb.Append("//     the code is regenerated.");
            sb.Append(Environment.NewLine);
            sb.Append("// </auto-generated>");
            sb.Append(Environment.NewLine);
            sb.Append("//------------------------------------------------------------------------------");
            sb.Append(Environment.NewLine);
            sb.Append(Environment.NewLine);
            sb.Append("using System.Collections.Generic;");
            sb.Append(Environment.NewLine);
            sb.Append("using UnityEngine.Localization.Settings;");
            sb.Append(Environment.NewLine);
            sb.Append(Environment.NewLine);
            sb.Append("namespace LurkingNinja.MyGame.Internationalization");
            sb.Append(Environment.NewLine);
            sb.Append("{");
            sb.Append(Environment.NewLine);
            sb.Append($"\tpublic static class i18n_{table.TableCollectionName}");
            sb.Append(Environment.NewLine);
            sb.Append("\t{");
            sb.Append(Environment.NewLine);
            sb.Append($"\t\tprivate const string NAME = \"{table.TableCollectionName}\";");         
            sb.Append(Environment.NewLine);
            foreach (var entry in table.Entries)
            {
                sb.Append($"\t\tpublic static string {entry.Key}(List<object> o = null) => LocalizationSettings.StringDatabase.GetLocalizedString(NAME, {entry.Id}, o);");
                sb.Append(Environment.NewLine);
            }
            sb.Append("\t}");
            sb.Append(Environment.NewLine);
            sb.Append("}");
            sb.Append(Environment.NewLine);
            return sb.ToString();
        }

But it would be great having a way to find out inside the SharedTableData.Entries that an entry is smart entry or not (any of the implementation is a smart entry or not).
I guess it is too late to restrict the entry keys to be valid C# identifiers? :slight_smile:

Interesting. You may find it easier to put the bulk of your code into a text template file and then just replace the content, a bit like how we generate the MonoBehaviour files etc.
The Smart Entry is a table-specific thing, so storing it in the Shared Table Data would not really serve many purposes, it would just mean duplicating the data and then having to keep it in sync. Could you generate the file using a StringTableCollection instead? That would let you query the table entries to see if they are Smart. Why do you need to know if they are Smart for this?

Haha yes and I don’t think it would work for everyone. I would run the Key through some sort of converter that removes invalid characters.

I know, it’s scheduled, I just haven’t got around to do that. It is a first concept, a thinking out loud code if you will. :slight_smile:

I know, and it works this way for anyone, who doesn’t try to handle the entries as one concept. Like me. In my concept all the entries appear once.
What is weird for me, that you decided to handle the entries separately. I mean semantically the English translation of KEY key and the Hungarian translation of KEY key should not mean different things. I can’t cite any situations where you want one translation to be different than the other. I understand that the actual values are in separate tables, data locality is important, they need to be loadable separately, but not their behavior: I don’t see any situation where one translation would be smart entry and any other language would not require the same.
Now, of course I can and I will have to iterate over all of the locales one by one and check if the same thing is smart entry or not, but it really shouldn’t be necessary.

Less error prone if I do not provide default value and actually throw an error if the usage of such entry needs a value passed in. Like I would generate a property for entries don’t need values and methods for entries do without default value.
It is better to catch problems like this in compile time than hunting the “{0}” texts on screen during manual testing. The manual test round-trips are expensive, having compile error right away is cheaper.

I figured, but I had to try. :smile: I know, I’m working on the regexp. It’s harder than it looks, if you care about the most freedom of choice. C# has a surprisingly permissive syntax.

1 Like

Okay, do you have any good entry point to catch this? Because in the OnAssetPostProcessor I get callback on SharedTableData when I edit any StringTables (and presumably any asset tables as well), but I do not get callback for the Collection, I’m guessing only when I add or remove entire tables.
The SharedTableData doesn’t seem to contain any data about its parent.
So how can I arrive to the Collection in any sensible manner? Unfortunately the new docs format isn’t great communicating concepts, only syntax.

I saw that we have some add/remove/modify/etc events somewhere. Do you think I could tap into these events from an editor script somehow?

When you get the callback for the asset being edited you could use LocalizationEditorSettings.GetCollectionForSharedTableData(SharedTableData) to get the collection. We do also have editor events as you said which will tell you when certain changes are made although GetCollectionForSharedTableData is probably enough. Just cast the result to a StringTable collection.

Thanks! Appreciate it! If someone interested, here’s what I ended up with for the time being:

using System;
using System.IO;
using System.Linq;
using System.Text;
using UnityEditor;
using UnityEditor.Localization;
using UnityEngine;
using UnityEngine.Localization.Tables;
using Object = UnityEngine.Object;

namespace LurkingNinja.MyGame.Editor
{
    public class OnAssetPostProcess : AssetPostprocessor
    {
        private const string PATH = "/MyGame/Code/_generated/i18n/";
        private const string TEMPLATE_PATH = "Assets/MyGame/Editor/i18n/i18n.cs.txt";
     
        private static string GetFullPath(string fileName) => $"{Application.dataPath}{PATH}{fileName}.cs";

        private static void GenerateTableAccessorFile(SharedTableData table, string fileName)
        {
         
            var genPath = GetFullPath(fileName);
            using var writer = new StreamWriter(genPath, false);
            writer.WriteLine(GenerateFileContent(table));
            AssetDatabase.ImportAsset($"Assets{PATH}{fileName}.cs");
        }

        private static string KeyToCSharp(string key)
        {
            if(string.IsNullOrEmpty(key))
                throw new ArgumentOutOfRangeException(nameof(key), "Translation key cannot be empty or null.");
            if (char.IsNumber(key[0])) key = $"_{key}";
            key = key.Replace(" ", "_");
            return $"@{key}";
        }

        private static bool IsSmart(StringTableCollection tableCollection, long id)
        {
            if(tableCollection == null) return false;
            return tableCollection.StringTables.Select(stable =>
                    stable.GetEntry(id)).Any(tableEntry => tableEntry.IsSmart);
        }

        private static string GenerateFileContent(SharedTableData table)
        {
            var tableCollection = LocalizationEditorSettings
                    .GetStringTableCollection(table.TableCollectionNameGuid);
            var template = AssetDatabase.LoadAssetAtPath<TextAsset>(TEMPLATE_PATH).text;
            var sb = new StringBuilder();

            foreach (var entry in table.Entries)
            {
                var key = KeyToCSharp(entry.Key);
                sb.Append($"\t\t\tpublic static string {key}");
                if(IsSmart(tableCollection, entry.Id)) sb.Append("(List<object> o)");
                sb.Append($" => LocalizationSettings.StringDatabase.GetLocalizedString(NAME, {entry.Id}, o);");
                sb.Append(Environment.NewLine);
            }
            return string.Format(
                template, DateTime.Now, KeyToCSharp(table.TableCollectionName), table.TableCollectionName, sb);
        }

        private static string GetFileName(string fileName) =>
                Path.GetFileNameWithoutExtension(fileName).Replace(" ", "_");

        private static void OnPostprocessAllAssets(
            string[] importedAssets, string[] deletedAssets, string[] movedAssets, string[] movedFromAssetPaths)
        {
            foreach (var path in importedAssets)
            {
                var obj = AssetDatabase.LoadAssetAtPath<Object>(path);
                if(obj is not SharedTableData tableData) continue;
                GenerateTableAccessorFile(tableData, GetFileName(path));
            }

            foreach (var path in deletedAssets)
            {
                var fileName = GetFileName(path);
                if (File.Exists(GetFullPath(fileName))) File.Delete(GetFullPath(fileName));
            }
        }
    }
}

i18n.cs.txt

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Generated: {0}
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

using System.Collections.Generic;
using UnityEngine.Localization.Settings;

namespace LurkingNinja.MyGame.Internationalization
{{
    public static partial class I18N
    {{
        public static class {1}
        {{
            private const string NAME = "{2}";
{3}
        }}
    }}
}}
//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Generated: 6/5/2021 5:35:36 PM
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

using System.Collections.Generic;
using UnityEngine.Localization.Settings;

namespace LurkingNinja.MyGame.Internationalization
{
    public static partial class I18N
    {
        public static class @MainMenu
        {
            private const string NAME = "MainMenu";
            public static string @VERSION(List<object> o) => LocalizationSettings.StringDatabase.GetLocalizedString(NAME, 20358541312, o);
            public static string @SINGLE => LocalizationSettings.StringDatabase.GetLocalizedString(NAME, 8068326141952);
        }
    }
}

And I have a partial class

using System.Collections.Generic;
using UnityEngine.Localization.Settings;
using UnityEngine.ResourceManagement.AsyncOperations;

namespace LurkingNinja.MyGame.Internationalization
{
    public static partial class I18N
    {}
}

Nice.
For checking the number at the start you could use

char.IsNumber(key[0])
1 Like

@ Thanks for your Scripts! I’m trying to use them with following set up:

Assets/Editor/LurkingNinja/I18N.cs.txt
Assets/Editor/LurkingNinja/OnAssetPostProcess.cs
Assets/Editor/LurkingNinja/I18N.cs (with partial class code)

and in OnAssetPostProcess for pathes

private const string PATH = "/Editor/LurkingNinja/";
private const string TEMPLATE_PATH = "Assets/Editor/LurkingNinja/i18n.cs.txt";

When I change a Key in the String Tables and save a file with name New_Table_Shared_Data is generated but also this error is thrown:

NullReferenceException: Object reference not set to an instance of an object
LurkingNinja.MyGame.Editor.OnAssetPostProcess+<>c.b__5_1 (UnityEngine.Localization.Tables.StringTableEntry tableEntry) (at Assets/Editor/LurkingNinja/OnAssetPostProcess.cs:42)
System.Linq.Enumerable.Any[TSource] (System.Collections.Generic.IEnumerable1[T] source, System.Func2[T,TResult] predicate) (at <61774763be294c9f8e2c781f10819224>:0)
LurkingNinja.MyGame.Editor.OnAssetPostProcess.IsSmart (UnityEditor.Localization.StringTableCollection tableCollection, System.Int64 id) (at Assets/Editor/LurkingNinja/OnAssetPostProcess.cs:41)
LurkingNinja.MyGame.Editor.OnAssetPostProcess.GenerateFileContent (UnityEngine.Localization.Tables.SharedTableData table) (at Assets/Editor/LurkingNinja/OnAssetPostProcess.cs:56)
LurkingNinja.MyGame.Editor.OnAssetPostProcess.GenerateTableAccessorFile (UnityEngine.Localization.Tables.SharedTableData table, System.String fileName) (at Assets/Editor/LurkingNinja/OnAssetPostProcess.cs:25)
LurkingNinja.MyGame.Editor.OnAssetPostProcess.OnPostprocessAllAssets (System.String[ ] importedAssets, System.String[ ] deletedAssets, System.String[ ] movedAssets, System.String[ ] movedFromAssetPaths) (at Assets/Editor/LurkingNinja/OnAssetPostProcess.cs:74)
System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[ ] parameters, System.Globalization.CultureInfo culture) (at <00c558282d074245ab3496e2d108079b>:0)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[ ] parameters, System.Globalization.CultureInfo culture) (at <00c558282d074245ab3496e2d108079b>:0)
System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[ ] parameters) (at <00c558282d074245ab3496e2d108079b>:0)
UnityEditor.AssetPostprocessingInternal.InvokeMethod (System.Reflection.MethodInfo method, System.Object[ ] args) (at <948074e677924ec29383d02747cdda34>:0)
UnityEditor.AssetPostprocessingInternal.PostprocessAllAssets (System.String[ ] importedAssets, System.String[ ] addedAssets, System.String[ ] deletedAssets, System.String[ ] movedAssets, System.String[ ] movedFromPathAssets, System.Boolean didDomainReload) (at <948074e677924ec29383d02747cdda34>:0)
UnityEditor.EditorApplication:Internal_CallGlobalEventHandler()

if my set up is right then it looks like there is an unhandled case for my String Table in the OnAssetPostProcess which causes this error. When I have Time later this day I’ll try your code in a new clean Project with a single entry String Table and give you feedback if that solved the error or if it still exists.

This shouldn’t point to the Editor, it needs to be part of the run-time if you’re using this for translating in-game text.

As for the NRE, I have no idea at first sight, I have (very) basic smart entries and they are working, so it is possible that I missed something if you have more complex cases.
Of course any feedback is welcome and appreciate it!

1 Like

Thx for you work, it saved a lot of my time !)

2 Likes