Creating StringTable at Runtime

Hi!
I wish to be able to import files (e.g., csv) into new StringTables and insert them into the StringDatabase during runtime.

I understand how it’s done through the editor but I can’t seem to figure out how to do this using only the UnityEngine.Localization namespace.

What I have managed so far:
Import the csvs files
Create and add new locale to LocalizationSettings
Create and add entries read from csv into a new StringTable

What I am still missing:
Get the SharedData table from the Database.
Add the new StringTable to the Database.

Thanks!

Hmm. Ok you would need to register the new tables with the LocalizedStringDatabase during play. The API for this is not public at the moment. Ill make a task to support user created StringTables in playmode.
One thing you could do now and that I would recommend is to create a custom addressables provider to provide the StringTable. This is the approach we will go with in the future when we support loading additional formats in player.

Look at ResourceProviderBase

This is great! I’ll look into it!

Thanks, @karl_jones !

I have created a IResourceProvider called CsvTableProvider and added as the first IResourceProvider in the ResourceManger providers by calling:

AddressableAssets.Addressables.ResourceManager.ResourceProviders.Insert(0, new CsvTableProvider(_sharedDataTable));

However, when I change to the newly added Locale, the LocalizedDatabase class calls Addressables.LoadAssetAsync<TTable>(tableAddress); and fails with four errors logged in the following order:

  • Exception encountered in operation CompletedOperation, status=Failed, result= : Exception of type ‘UnityEngine.AddressableAssets.InvalidKeyException’ was thrown., Key=UI Text_en-GB, Type=UnityEngine.Localization.Tables.StringTable
  • Failed to load table: CompletedOperation
  • Exception: Exception of type ‘UnityEngine.AddressableAssets.InvalidKeyException’ was thrown., Key=UI Text_en-GB, Type=UnityEngine.Localization.Tables.StringTable
  • Exception encountered in operation CompletedOperation, status=Failed, result=UnityEngine.Localization.Settings.LocalizedDatabase`2+TableEntryResult[[UnityEngine.Localization.Tables.StringTable, Unity.Localization, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null],[UnityEngine.Localization.Tables.StringTableEntry, Unity.Localization, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]] : Failed to load table: CompletedOperation

Here is the CsvTableProvider implementation:

public class CsvTableProvider : ResourceProviderBase
{
    private SharedTableData _sharedDataTable;

    public override string ProviderId => "UnityEngine.ResourceManagement.ResourceProviders.AssetDatabaseProvider";


    public CsvTableProvider(SharedTableData sharedDataTable)
    {
        _sharedDataTable = sharedDataTable;
    }

    public override void Provide(ProvideHandle provideHandle)
    {

        string filePath = ResourceLocationCsvPath(provideHandle.Location);

        StringTable st = ImportTable(filePath);

        object result;
        if (provideHandle.Type.IsArray)
        {
            result = ResourceManagerConfig.CreateArrayResult(provideHandle.Type, new UnityEngine.Object[]{ st});
        }
        else if (provideHandle.Type.IsGenericType && typeof(IList<>) == provideHandle.Type.GetGenericTypeDefinition())
        {
            result = ResourceManagerConfig.CreateListResult(provideHandle.Type, new UnityEngine.Object[] { st });
        }
        else
        {
            result = st;
        }


        provideHandle.Complete(result, result != null, result == null ? new Exception($"Unable to load asset of type {provideHandle.Type} from location {provideHandle.Location}.") : null);
    }


    public override bool CanProvide(Type t, IResourceLocation location)
    {
        string path = ResourceLocationCsvPath(location);

        return
            t.Equals(typeof(StringTable))
            && (path != null);
    }


    private StringTable ImportTable(string filePath)
    {
        Locale newLocale = Locale.CreateLocale(Path.GetFileNameWithoutExtension(filePath));
        List<(string key, string value)> entries = ReadCsvEntries(filePath);

        StringTable stringTable = ScriptableObject.CreateInstance<StringTable>();
        stringTable.LocaleIdentifier = newLocale.Identifier;
        stringTable.SharedData = _sharedDataTable;


        for (int i = 0; i < entries.Count; i++)
        {
            stringTable.AddEntry(
                entries[i].key,
                entries[i].value);
        }


        return stringTable;
    }

    private List<(string, string)> ReadCsvEntries(string filePath)
    {

        List<(string, string)> entries = new List<(string, string)>();
        DataTable csvTable = new DataTable();
        using (var stream = new StreamReader(filePath))
        {
            using (var csvReader = new CsvReader(stream, CultureInfo.InvariantCulture))
            {
                using (var dr = new CsvDataReader(csvReader))
                {
                    csvTable.Load(dr);


                    for (int i = 0; i < csvTable.Rows.Count; i++)
                    {
                        entries.Add(
                            (csvTable.Rows[i][0] as string,
                            csvTable.Rows[i][1] as string)
                        );
                    }
                }
            }

        }
        return entries;
    }

    private string ResourceLocationCsvPath(IResourceLocation location)
    {
        if(_sharedDataTable.TableCollectionName.Length >= location.PrimaryKey.Length)
            return null;

        string fileName = location.PrimaryKey.Substring(_sharedDataTable.TableCollectionName.Length + 1); //Expected StringTable PrimaryKey format: "{Collection.Name}_{CultureName}"
        string filePath = Path.Combine(Application.streamingAssetsPath, fileName + ".csv");

        return File.Exists(filePath) ? filePath : null;

    }

}

I have manage to do it but there is still some concepts (related to Addressables) that are not clear to me.
Before, I’ve completely missed the IResourceLocator interface concept. That is why I did the ProviderId atrocity above.

From creating my own IResourceLocator, I managed to set the ProviderId in CsvTableProvider as well as guarantee that my Location will have a Locator. (This is what caused the four exceptions from above).

In case anyone wonders, below is a simple implementation of an IResourceLocator

public class CsvResourceLocator : IResourceLocator
{
    public string LocatorId => nameof(CsvResourceLocator);

    public IEnumerable<object> Keys => new object[0];

    public bool Locate(object key, Type type, out IList<IResourceLocation> locations)
    {
        if (!typeof(StringTable).IsAssignableFrom(type))
        {
            locations = null;
            return false;
        }

        locations = new List<IResourceLocation>();
       
        IResourceLocation[] noDependencies = new ResourceLocationBase[0];
        locations.Add(new ResourceLocationBase(key as string, key as string, typeof(CsvTableProvider).FullName, type, noDependencies));
       

        return true;
    }

if anyone could point out any misconceptions from this snippet, I would be grateful. I have no idea what some parameters in ResourceLocationBase are for :slight_smile:

I’m not too familiar with this area at the moment. I’ll ask the Addressables teram if they have any advice.

One thing I think you will need to do is add the locator and provider to Addressables:

IEnumerator Start()
    {
        yield return Addressables.InitializeAsync();

        Addressables.ResourceManager.ResourceProviders.Add(new CsvTableProvider(null));
        Addressables.AddResourceLocator(new CsvResourceLocator());
    }

Once I did this it started to work for me.

Edit:

It looks like what you have works pretty well now. Im going to pin this thread as it seems like something others will want to do and it’s a good place to start :slight_smile:
We do plans to have support various custom providers and locators in the future so people can add modding support, pull google sheets in the player etc.

1 Like

Hi,
I am currently trying to create a runtime translation updater in my application based on a CSV file downloaded from the server.
The CSV file is generated on the server and has identical structure to that exported in the CSV Extension available from the String Table Collections object.

I tried to use the script suggested by you but I totally don’t know how to initialize it and upload CSV file - I suspect I need to run Provide (ProvideHandle provideHandle) but I don’t know how to create ProvideHandle object with information about CSV file. In addition, it seems to me that the above script is used to add a new Locale with values.
I will definitely need it in the future, but at the moment I am looking for a solution to update and save translations in the runtime.

I used and tweaked a little bit the above class for CSV interpretation and creation of temp string table and wrote something like this:

public IEnumerator UpdateTranslations(string _translationsCSVFilePath){
       
        // save starting locale
        Locale _baseLocale = LocalizationSettings.SelectedLocale;
 
        // loop throu all available locales
        foreach(Locale _locale in LocalizationSettings.AvailableLocales.Locales){
   
          
LocalizationSettings.SelectedLocale = _locale;

            // get localized string table database based on active locale
            AsyncOperationHandle<StringTable> _asyncStringTableDatabase =                                  LocalizationSettings.StringDatabase.GetTableAsync(LocalizedStringTables.TableReference);
            yield return new WaitUntil(()=> _asyncStringTableDatabase.IsDone);
            StringTable _stDatabase = _asyncStringTableDatabase.Result;
          
            // create string table corresponding to active locale (CSV contains cols from all locales).
            // The safest way to get propper CSV structure is to use export CSV extension from your primary String Table Collection object in Unity inspector
            StringTable _importedST = ImportTable(_translationsCSVFilePath, _locale);

           
// this part finds and adds fields that are empty in selected language string database but exist in shared values and in imported StringTable. 
            // If fields in database are empty they can't be updated - _stDatabase.Values.Count = number of filled fields
            // further more it will add new row in Localization Tables if the key is unique - it desn't exist
            foreach(StringTableEntry _value in _importedST.Values){
                if(_value.Value != "" && _value.Value != null){
                    if(_stDatabase.SharedData.Contains(_value.KeyId)){
                        if(!_stDatabase.ContainsKey(_value.KeyId)){                                                
                            _stDatabase.AddEntry(_value.Key, _value.Value);                        
                        }
                    }
                }          
            }

            // this part updates the editor runtime string database values - will be visable in runtime. Also the Window > Asset Management > Localization Talbes, will also be updated but after you stop and run the app again from the editor
            foreach(StringTableEntry _value in _stDatabase.Values){
              
                if(_importedST[_value.Key] == null){ continue;} // for safety - skip that translation part if key from database doesnt exist in imported csv
                string _val = _importedST[_value.Key].Value;                          
              
                if(_value.Value != _val){
                    _value.Value = _val;
                }              
            }          

            // get addressable string tables based on active locale. Just pulling the values will update displayed values on the device
            AsyncOperationHandle<StringTable> _asyncAddressablesTable =  Addressables.LoadAssetAsync<StringTable>("Locale-" + _locale.Identifier.Code);
            yield return new WaitUntil(()=> _asyncAddressablesTable.IsDone);          
            StringTable _addressablesTable = _asyncAddressablesTable.Result;

            // This script will work only if in "Window > Asset Managmenet > Addressables > Groups" window, the "Simulate Groups (advanced)" option located in the "Play Mode Script" dropdown will be selected.
   
        }

        // switch to saved starting locale
        LocalizationSettings.SelectedLocale = _baseLocale;

        AppManager.SetBaseMessages();
    }



    private StringTable ImportTable(string _filePath, Locale _locale)
    {
        // Locale newLocale = Locale.CreateLocale(Path.GetFileNameWithoutExtension(_filePath));
        List<(string _key, string _value)> _entries = ReadCsvEntries(_filePath, _locale.LocaleName);
        StringTable _stringTable = ScriptableObject.CreateInstance<StringTable>();
        _stringTable.LocaleIdentifier = _locale.Identifier;
        _stringTable.SharedData = SharedTable;

 
        for (int i = 0; i < _entries.Count; i++)
        {
            _stringTable.AddEntry(
                _entries[i]._key,
                _entries[i]._value);
        }
        return _stringTable;
    }

    private List<(string, string)> ReadCsvEntries(string _filePath, string _localeName)
    {
        List<(string, string)> _entries = new List<(string, string)>();
        DataTable _csvTable = new DataTable();
        using (var _stream = new StreamReader(_filePath))
        {
            using (var _csv = new CsvReader(_stream, CultureInfo.InvariantCulture))
            {
       
                _csv.Read();
                _csv.ReadHeader();
                while (_csv.Read())
                {
                    _entries.Add((
                        _csv.GetField("Key"),
                        _csv.GetField(_localeName.Replace(") (", ")("))));
// it mightr be a bug but the  Locale identifier and he header in exported CSV file corresponding to the language differs - theres extra space between brackets
// standard CSV Extension export -> English (United Kingdom)(en-GB), Locale English (United Kingdom) (en-GB)
// the CSV names can be changed in CSV Extension, but keep in mind that the standard extension removes this space (I didn't notice it at first)
                }
            }
        }
        return _entries;
    }

as you will notice the CSV file is used to create temporary StringTable instances with the Key, and value columns coresponding to the currently selected Locale. Then a few parts that update the values according to the created table - the values displayed in the editor player, the stringtables database (which is useful so that you do not have to do it manually each time) and AddressableTables, which I care about the most, i.e. changing the values on users’ devices.
However, I have a problem - updating values on users’ devices only works if we run the above script. After restarting the application, the values return to their original values.
I suspect it has something to do with updating the Catalogs, but when I run a script that I found on another post:

private IEnumerator UpdateCatalogs()
    {
        List<string> catalogsToUpdate = new List<string>();
        AsyncOperationHandle<List<string>> checkForUpdateHandle = Addressables.CheckForCatalogUpdates();
 
        checkForUpdateHandle.Completed += op =>
        {
            Debug.Log(op.Result.Count);
            catalogsToUpdate.AddRange(op.Result);
        };
        yield return checkForUpdateHandle;
        if (catalogsToUpdate.Count > 0)
        {   
            AsyncOperationHandle<List<IResourceLocator>> updateHandle = Addressables.UpdateCatalogs(catalogsToUpdate);
            yield return updateHandle;
        }
    }

…it does not find any values to update.
What am I doing wrong? How can I save modified values and StringTables in runtime on the device (not only in editor player runtime) so the next time i call Localized string i’ll get the updated values?

Changes you make to a table at runtime wont persist after the application restarts, they are just in memory and will be lost when the application closes. Addressables does have a system for content updates: https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/ContentUpdateWorkflow.html

For what you are doing your script will always need to run in order to patch the String Table, this is fine to do. If the problem is that you need to pull a csv file from your server each time then it may be worth trying to cache the csv file locally, such as un PlayerPrefs and then only pull the table if you dont have one cached or every so often.

Thanks for quick reply

Well, I can always save a file to disk (actually, now I do this before using a CSV file) and download it again only when I require it.
However, I was hoping to somehow skip the entire process of creating tables and updating the values each time the application starts - it is only a second or so, but each second is important…

From what I read in the link mentioned, it is also possible to update by building remote directories, using external RemoteLoadPath.

7468241--917372--aaa.jpg
I will also have to try it, but at first glance it seems to me that this method is not very optimal (i might be wrong) … safe because the groups are generated by the unity editor, but time and resource intensive if there are many languages and even more phrases to update…
Correct me if I’m wrong but from what I understand is that every time, apart from updating the values in StringTablesDatabase, I will also have to build AddressableGroups and push them to the server, these groups then will be retrieved from the server by the app each time the aplication starts and updated if I use Addressables.CheckForCatalogUpdates() ?

Im afraid I dont know the details about how Addressables does its checking, you should be able to get the answer in the Addressables forum section Unity Engine - Unity Discussions

Ok, thanks for the advice,
However, I think I’ll go with serializing and deserializing tables generated from CSV to speed up the process.
cheers

1 Like

Hi, you referred me to here, I just had a look, would you be able to explain how addressables work? What is the CsvTableProvider? And the CsvResourceLocator? And how is this creating a new string table? Is that what this is doing? lol, sorry I’m so confused. I feel dumb today lol

Addressables is the system Localization uses to fetch data such as String Tables. By default it comes from Asset Bundles but you can create custom providers to get data from other sources such as Csv, Json etc.
https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/index.html

I’m looking for a solution that does the same thing as what you’re looking for.
Can you find something relevant to make your StringTable update persistent in your app ?

Have you looked into content updates with addressables? These are persistent.
https://docs.unity3d.com/Packages/com.unity.addressables@1.19/manual/ContentUpdateWorkflow.html

Hi Karl,
This documentation is pretty dense.
I’m not sure how to apply what I understand there to the Localization system.
Can you tell me which part I should lean on?

Thank you so much !

1 Like

We have now released 1.4.2 which includes ITableProvider and ITablePostProcessor. These let you provide tables from custom locations and apply changes to a table when it first loads. These are a much simpler way than creating a custom addressables resource provider.

3 Likes

Hi,
regarding the docs of ITableProvider, it says it is useful for example for mods, however the example references
LocalizationEditorSettings from UnityEditor so that won’t work for loading a new table for a mod.
In our case mods typically don’t override base game content so having a separate table makes much more sense.
It would be great if there was a snippet there for that use case as well.

I see now why I am having issues with it.
In our case we have two sources of data, the addressable and a dynamic source for additional tables during runtime.
I guess the system is not designed for directly so I’ll have to figure out a workaround.

Ah, the example should not be using LocalizationEditorSettings. That’s a mistake. You can change that part to

var settings = LocalizationSettings.Instance

Ill get the example fixed.

Can you provide some more details? The system should be able to handle this.