Version 1.4.1-Preview 1 with Dynamic SDF for Unity 2018.3 now available!

[Stephan_B]

At long last, the wait is over!

Today, I am pleased to announce the release of the first preview of TextMesh Pro version 1.4.0 for Unity 2018.3 with support for Dynamic SDF.

One of the most requested feature since the initial release of TMP has been support for Dynamic SDF which is now available in this new release.

Dynamic SDF makes it possible to add glyphs and characters to font assets at runtime thus significantly enhancing and simplifying workflows for handling localization and languages like CJK with large character sets. This new functionality is supported throughout TMP and its fallback system.

In order to support this new feature, structural changes to Font Assets and Sprite Asset have been necessary and will require existing font assets and sprite assets to be upgraded. This upgrade should be automatically handled by the system as these resources are loaded / accessed in your projects. Although I have been testing this process using dozens of user projects where everything worked fine, I am releasing this new version as a preview as I need your help in further testing and making sure this process works as expected. With that said …

Steps to Upgrade to this new release
(1) Backup your project first. This is a preview release so let’s not take any chances.
(2) Open this backup of your project in Unity 2018.3.
(3) Close any scene that may already be opened in Unity. Although the asset migration process should still work. It is preferable to close any opened scenes.
(4) Open Package Manager UI via the “Window - Package Manager” menu and as per the image below select the latest preview version available for version 1.4.0.

(5) Start opening scenes to see if everything works as expected. If not please report these issues in this thread and as usual I’ll get those addressed.

(6) If your project contains many large font assets where many of them happen to be assigned as fallbacks in the TMP Settings, opening the TMP Settings which would force all these font assets to converted as a group might appear to lock up the editor. So if you run into this, instead of accessing the TMP Settings, just select each of these font assets one at a time in the project tab. Wait for each of them to be converted before selecting the next one.

This should be it in terms of the migration.

Next is getting familiar with the new Dynamic SDF system and added flexibility it will offer and new potential workflows in regards to managing these resources. I have already published the following video which covers a big chunk of the new Dynamic System and Fallback.

The following new video covers the upgrade process as well as provides insight on the structural changes to Font Assets and Sprite Assets. The video also covers some new additional functionality that had not previously been shown / discussed.

TMP Release 1.4.0 Upgrade Process

FAQ
Question 1
Can the content of a font asset be cleared / reset?

Yes. In the Editor, you can use the Context Menu option Reset which will clear the Character, Glyph and Glyph Adjustment Tables as well as reset the atlas texture back to size 0.

Via scripting you can use the following function.

public void ClearFontAssetData(bool setAtlasSizeToZero = false)

Question 2
I am trying to switch a previously created font asset to dynamic mode and getting a warning about the atlas texture not being readable. How do I set the texture to readable?

In the font asset inspector, double click on the Font Atlas to navigate to the atlas texture. Then switch the inspector to debug mode where the Is Readable flag will now be exposed.

Question 3
Can a font asset be created at runtime?

Yes. The font asset can be created using the following function.

public static TMP_FontAsset CreateFontAsset(Font font, int samplingPointSize, int atlasPadding, GlyphRenderMode renderMode, int atlasWidth, int atlasHeight, AtlasPopulationMode atlasPopulationMode = AtlasPopulationMode.Dynamic)

As more questions or suggestions come up, I’ll add those to the FAQ.

Feedback
Please report any issues in this thread. Feel free to ask questions if anything is unclear. I am here and available to help.

[Fachmah]

Loving the update! Everything is working really well so far, however I have one question. Is it possible to make a dynamic font file overwrite itself when it has reached max capacity? For instance having a new character overwrite the first character that was added to the atlas.

Thanks again for the preview - much appreciated!

[Stephan_B]

@Fachmah

Loving the update! Everything is working really well so far, however I have one question. Is it possible to make a dynamic font file overwrite itself when it has reached max capacity? For instance having a new character overwrite the first character that was added to the atlas.

Thanks again for the preview - much appreciated!

Glad it is working well and you are most welcome

In terms of the question, the challenge with overwriting a specific glyph is that glyphs come in all shapes and sizes and as such the new glyph could be bigger than the one it is replacing. So doing this could be tricky.

Having said that, Font Assets can now be Dynamic in terms of adding glyphs and characters to them at runtime but Font Asset themselves can be created at runtime.

// Create a new font asset at runtime.
m_DynamicFontAsset = TMP_FontAsset.CreateFontAsset(SourceFontFile, 90, 9, GlyphRenderMode.SDFAA, 1024, 1024, AtlasPopulationMode.Dynamic);

// Force some characters to be added to the font asset. Ie. as opposed to some text object forcing those to be added).
m_DynamicFontAsset.TryAddCharacters("1234");

//  Assign newly created font asset to some text object.
m_TextComponent.font = m_DynamicFontAsset;

// Set the text on the object which in turn will force these to be added to this new font asset dynamically.
m_TextComponent.text = "ABCD";

Given the above code and assuming those characters were available in the font file, the font asset and atlas texture would end up with “1234” and “ABCD” in it.

You can also clear the font asset data using the following. This clears the glyph, character and glyph adjustment tables as well as atlas texture (reset to 0 X 0 in size) but maintains all the relevant settings so you can use it normally and resume adding characters to it.

m_DynamicFontAsset.ClearFontAssetData();

Between all of these, you can create new font assets, pre-populate them with some characters. Reset them to clear out their characters and all of that at runtime.

[BBO_Lagoon]

Hi Stephan,

First, thanks for all your work !
I’m really happy with the dynamic generation.

There is a problem still.
With texts that are animated (with scale), lot of time on my UIs I have texts that are scaled to 0 and then show with a tween from scale 0 to 1. These texts show as white squares. Texts that start with small scale but not 0 are blurry.
If I disable / enable the TextMeshPro component all goes weel.

Would be good to fix that.

[Stephan_B]

@BBO_Lagoon

Hi Stephan,

First, thanks for all your work !
I’m really happy with the dynamic generation.

There is a problem still.
With texts that are animated (with scale), lot of time on my UIs I have texts that are scaled to 0 and then show with a tween from scale 0 to 1. These texts show as white squares. Texts that start with small scale but not 0 are blurry.
If I disable / enable the TextMeshPro component all goes weel.

Would be good to fix that.

Most of the time this issue is the result of the scale / tweening not doing uniform scaling on the X, Y and Z. For certain, a scale of zero on the Y of the text object would cause the behavior you describe.

Note that I did change the handling of the SDF scaling so if the issue persists, I’ll take a look at it and if you could please provide some simple repro project / scene that I could use to reproduce the behavior which usually greatly speeds up the time it takes to resolve this issues.

[Fachmah]

@Stephan_B

Glad it is working well and you are most welcome

In terms of the question, the challenge with overwriting a specific glyph is that glyphs come in all shapes and sizes and as such the new glyph could be bigger than the one it is replacing. So doing this could be tricky.

This makes a lot of sense, I guess I’ll work in the ClearFontAssetData() instead. It works the same way as pressing reset in the inspector would right? :–)

[Stephan_B]

@Fachmah

This makes a lot of sense, I guess I’ll work in the ClearFontAssetData() instead. It works the same way as pressing reset in the inspector would right? :–)

It does indeed. I use it myself when I need to reset / clear a font asset for testing dynamically adding characters in builds / on devices.

Now just to make sure I don’t overlook anything with respect to your use case, what was your thought with regards to doing it one character at a time?

P.S. Since it is already 4:00 on my end, I am going to go sleep but I’ll follow up tomorrow.

[BBO_Lagoon]

Here is the repro project. You can see the problem appear with TextMeshPro 1.4 but not with the 1.3

4171111–368452–TMP_1.4.0_ScaleIssue.rar (1.32 MB)

[Stephan_B]

@BBO_Lagoon

Here is the repro project. You can see the problem appear with TextMeshPro 1.4 but not with the 1.3

Thank you. I’ll take a look tomorrow when I wake up.

Update - I know what the issue is and will have a fix / code change you can make tomorrow.

[Stephan_B]

@BBO_Lagoon Thanks for the time you took to prepare the Repro project as it allowed me to uncover another issue which I would not have otherwise.

The following changes will be necessary to resolve the issue.

Let me know if this resolves the issue. If so these changes will be included in the new preview release.

Special Note: As of Unity 2018.3, packages are additionally cached on a per project basis where these local cached version are located in the project folder in “Library/PackageCache/…” which is where these changes should be done for the given project.

[LeeSinLcj]

Hello Stephan,I’m very glad to use the dynamic font system,but I found a GC.Alloc issue when using it.

As you can see, this method eat 2.3MB heap memory per call, and when I try to type some chinese word in the IuputField, memory grow up very fast.I notice that in your code there is a TODO work,

so could you fixed it or give me some advice?Thank you very much!!!

[Stephan_B]

@LeeSinLcj

Hello Stephan,I’m very glad to use the dynamic font system,but I found a GC.Alloc issue when using it.
As you can see, this method eat 2.3MB heap memory per call, and when I try to type some chinese word in the IuputField, memory grow up very fast.I notice that in your code there is a TODO work, so could you fixed it or give me some advice?Thank you very much!!!

By the time I got around to adding the ability to retrieve Glyph Adjustment Data (Kerning) dynamically, it became clear that I was going to need to make changes to the FontEngine and bindings which unfortunately are part of Unity itself and not part of the TMP package. As such this will require a Unity update which I should be able to get approved for 2019.1 but unlikely for 2018.3 given focus now is all on stability.

As per my TODO note, I am fully aware of the shortcomings of that current implementation and not happy about it but had limited options if I wanted this functionality short term.

Having said that, I could add new settings to the TMP Settings where trying to retrieve Glyph Adjustment Data dynamically at runtime could be enabled / disabled.

For some languages Kerning is pretty common but for CJK not as much so maybe this could be a viable short term solution.

P.S. Can you give me an idea of the number of glyphs your font asset contains?

[LeeSinLcj]

@Stephan_B

By the time I got around to adding the ability to retrieve Glyph Adjustment Data (Kerning) dynamically, it became clear that I was going to need to make changes to the FontEngine and bindings which unfortunately are part of Unity itself and not part of the TMP package. As such this will require a Unity update which I should be able to get approved for 2019.1 but unlikely for 2018.3 given focus now is all on stability.

As per my TODO note, I am fully aware of the shortcomings of that current implementation and not happy about it but had limited options if I wanted this functionality short term.

Having said that, I could add new settings to the TMP Settings where trying to retrieve Glyph Adjustment Data dynamically at runtime could be enabled / disabled.

For some languages Kerning is pretty common but for CJK not as much so maybe this could be a viable short term solution.

P.S. Can you give me an idea of the number of glyphs your font asset contains?

We are using Source Hans Sans now and it contains over 60000 glyphs.

[Stephan_B]

@LeeSinLcj

We are using Source Hans Sans now and it contains over 60000 glyphs.

How many glyphs are you planning to add / does your font asset contain?

For any given project / per language set, you are likely to only end up using a very small subset of those 60,000 glyphs.

My recommendations remain to have your Primary font assets (for any given language or groups of languages) contain the known characters / glyphs in the project which are the characters found in your the your titles, menus, ui, etc. This font asset should be prepared ahead of time just like before and be static.

Then to handle user input and all other potential characters, assign to your primary static font asset a fallback that is dynamic and there to catch all other (unknown / not already part of the project) characters.

This will provide for best performance and quality for all the known characters / text in the project. The primary would already contain the Glyph Adjustment Data (Kerning) since it is static and prepared ahead of time. Then for all other characters, where dynamic fallbacks are used there will be a higher performance overhead but your likely to only have to handle a small number of characters and furthermore given humans don’t type that fast relative to computer processing speed, the performance overhead will not be noticeable by any users.

Although you could rely on the dynamic system to handle all characters without preparing / using any static font assets, I don’t think that’s the best strategy for managing these resources.

P.S. Let me know if you feel adding the ability to disable dynamic retrieval of Glyph Adjustment Data in the TMP Settings would be useful?

[LeeSinLcj]

@Stephan_B

How many glyphs are you planning to add / does your font asset contain?

For any given project / per language set, you are likely to only end up using a very small subset of those 60,000 glyphs.

My recommendations remain to have your Primary font assets (for any given language or groups of languages) contain the known characters / glyphs in the project which are the characters found in your the your titles, menus, ui, etc. This font asset should be prepared ahead of time just like before and be static.

Then to handle user input and all other potential characters, assign to your primary static font asset a fallback that is dynamic and there to catch all other (unknown / not already part of the project) characters.

This will provide for best performance and quality for all the known characters / text in the project. The primary would already contain the Glyph Adjustment Data (Kerning) since it is static and prepared ahead of time. Then for all other characters, where dynamic fallbacks are used there will be a higher performance overhead but your likely to only have to handle a small number of characters and furthermore given humans don’t type that fast relative to computer processing speed, the performance overhead will not be noticeable by any users.

Although you could rely on the dynamic system to handle all characters without preparing / using any static font assets, I don’t think that’s the best strategy for managing these resources.

P.S. Let me know if you feel adding the ability to disable dynamic retrieval of Glyph Adjustment Data in the TMP Settings would be useful?

Thanks for your advice，I will try to use both static font and dynamic font, but I’m afraid that it could also cause memory leak issues when dynamic character was been created.

[Stephan_B]

Fix an issue where creating Font Assets at runtime were not getting their version number set correctly to version 1.1.0.

This change will be included in the next preview release but just in case, you wish to experiment with font asset creation at runtime, here are the relevant changes.

Special Note: As of Unity 2018.3, packages are additionally cached on a per project basis where these local cached version are located in the project folder in “Library/PackageCache/…” which is where these changes should be done for the given project.

[Stephan_B]

@LeeSinLcj

Thanks for your advice，I will try to use both static font and dynamic font, but I’m afraid that it could also cause memory leak issues when dynamic character was been created.

Adding a new character and glyph to a font asset will result in an allocation of 64 bytes for the glyph and 40 bytes for the character. There is also another allocation of 76 bytes for an internal list which I will try to re-work / remove. So in the next release version 1.4.0-preview.2 (which I’ll release this week) the one time allocation per character & glyph added to a font asset will be 180 bytes and if I can do away with the internal list that will reduce it by 76 bytes.

I have already added the ability to disable retrieving Glyph Adjustment Data (Kerning Pairs) dynamically. This option will be in the TMP Settings. I have already re-worked / removed the use of .ToArray() which was responsible for the bulk of those allocations. The revising implementation will be in another TMP package release which I should be able to get into 2019.1 and will do my best to get in 2018.3. Regardless I’ll look for some alternative for 2018.3 just in case.

Allocations per Glyph Pair Adjustment record is 64 bytes. So if you add a character & glyph that has 10 pairs then it will allocate 640 bytes.

Unless you are adding new characters, glyphs and adjustment pairs there should be no allocations.

[BBO_Lagoon]

Hi Stephan,
Thanks for the fix, it works now

[Stephan_B]

Glyph Adjustment Table Changes
In preparation for the addition of support for OpenType Font Features, the Glyph Adjustment Table presentation and data structure has been changed as seen below.

The updated Glyph Adjustment Table now shows the glyphs for each of the Glyph Pair Adjustment Records. Glyphs will only be shown if they are already present in the Glyph Table.

Individual pairs now reference the glyph index as opposed to unicode characters. When adding new adjustment pairs, the Character or its UTF16 or UTF32 representation can be used to lookup the glyph index for the given character.

Note that it is now possible to define / add GlyphPairAdjustmentRecords for glyphs that are not already present in the Glyph Table. This can occurs when the new “Get Font Features at Runtime” option is enabled in the TMP Settings.

When this option is enabled and as new glyphs are added to the Glyph Table, Glyph Pair Adjustment Records for those new glyphs will be automatically added to the Glyph Adjustment Table.

Font Assets that contain existing legacy kerning data will be automatically updated to the new format when these font assets are first loaded in the Editor or at Runtime (if somehow these font assets never ended up being loaded in the Editor during development or testing).

This new change will be included in the next preview release of TextMesh Pro package. This new release will be version 1.4.0-preview.2 which I am in the final stages of testing.

[evilock]

With the dynamic font, what would be the recommended way to manage the situation where the atlas reaches it’s maximum capacity? My use case would be an app where the user can just keep viewing content (in Chinese), gradually filling up the atlas with new glyphs, potentially up to the point that no new ones can be added. Even as a workaround, would there be a way to detect when this happens, so that we could then clear the dynamic atlas?