TMP: Breaking Chinese Bugs for Traditional ZhuYin ("BoPoMoFo" - Taiwan Locale)

Apologies for the spam, but replying on a new block for a breakthrough with a minimal repro ( @Stephan_B ) for 2 bugs:

  1. <u>bug</u> during character formation is caused by rich text:


Important: The expected fix is to still underline the text when character crafting (I don’t know the official term for this; whenever you’re mid-selection with the popup menu). However should not be used.

It seems like IME selection wraps with a . Instead, it should be explicitly underlined instead of using a wrapper.

  1. The stacking placeholder text with IME selection text:

Whenever you start “character crafting”, the placeholder text correctly hides. However, as soon as I alt+tab and return, the unfinished character crafting is not detected and stacks with the placeholder text.

EDIT 1: Unicode has a ton of weirdness that I can’t yet explain beyond even these. I’ll get back to you. Eg, this only happens with Unicode:

EDIT 2: One thing is for certain: Unity needs to fund you a Chinese IME QA tester. While I’m cool submitting these to you, it’s not very fair to the gamedev’s end-users that shouldn’t have to wait for fixes for common issues [common to a Chinese user] that shouldn’t exist. Since neither of us are Chinese, it’s sort of hard to come across real bugs. Unity claims full Chinese support – let’s hope they give you some +QA to help reinforce this claim (assuming you’re just one person patching TMPro).

More related repro:

If you type in the TMP_Input text directly (prefill within the editor) and FINISH your character (press 1 or spacebar at the suggestion popup), you get an extra character duplicated outside the wrapper:

This actually gets confusing very fast, so while there’s a static screenshot below, here’s a GIF showing multiple combos. To test, I’d just add Traditional Chinese (Taiwan) PinYin IME and try a few combos. In the example used, just type “xue”.

*The GIF above represents what happens both playing AND in the editor with the wrapping with the exception of the “extra” dupe character on the right that’s only found in the editor.

Expected: “學”
Actual: “學學”

I will be watching this thread daily: We’re now in Chinese beta, so Unicode bugs are spewing out like a fountain and happy to work with you to get these fixed as soon as possible.

Just wanted to follow up on this breaking issue: I should hopefully have provided sufficient info to resolve it. Seems like it’s simply a rich text underlining at IME character suggestion popups that should manually underlined (or ignored completely as a temporary workaround if rich text is disabled). Any luck?

May have found it:

As a temporary workaround, I’m going to guess I can comment this out? Or even better workaround would be to check if rich text is enabled (m_TextComponent.richText?) since the line shows normally when that happens. I’ll look around.

So as for the REAL fix, it would be some type of “override” where your (in this particular instance) would underline it no matter what, but not treated as rich text. This way, users can still enter input with !richText and still see IME underlines.

That temporary workaround did the trick:

5501548--564199--upload_2020-2-20_22-25-44.png

Note that there is no longer a <u></u> wrapper around the input caret when IME character is being selected.

Here is the code:

// TMP_InputField.cs:3093

// [OLD] This is bugged out when rich text is !enabled
//if (compositionLength > 0 && m_ReadOnly == false)

// [NEW] Only wrap <u></u> with richText enabled:
if (compositionLength > 0 && m_ReadOnly == false && m_TextComponent.richText)

See screenshot in previous bump for further reference.

I’m surprised this wasn’t reported ages ago by the Chinese community~ Are they using the old text system or something? We got bombed with reports from quite literally 30% of our entire Chinese community almost instantly with the Unicode issues.

Do you guys have a Unicode tester trying out all kinds of combos?

I am Chinese, and I was also surprised that Unity didn’t get bombed with bug reports regarding this issue, given that how big the Chinese user base is. Hearthstone is also developed using Unity, the desktop version of Hearthstone also has this bug. I don’t expect that we can find any perfect workaround. Although Stephan_B had said that he had resolved the issue, but the issue still exists in 2019.3. If we are using Unity engine to create Chinese word game and it’s on desktop platform, don’t use Unity InputField as the core gameplay, it’s unusable in Chinese.

I am aware of the issue when rich text is disabled. I just need to find the time to figure out a good solution for it. Until then, it might make sense to disable this underlining which is causing issues.

Note that I am reading all these posts but like I said before don’t always have time to reply but I am reading so keep the information / feedback coming.

2 Likes

Haha we have exactly that: An online text-based social game ;D We’ve implemented my workarounds mentioned above and it works, but it can be easy to get “lost” in your IME without an underline being there. At least it works, though. Surprised Hearthstone couldn’t find a fix. For deck names or something?

Now to remember I embedded the package for future fixes …

HI, I have the same issue,has it been fix? my version is 2.1.0-preview.8.

The and should no longer be visible when disabling Rich Text on the Input Field. It is still possible to get that while editing the IME composition. If that remains an issue, the following lines of code can be changes in your global package cache until I have a more permanent fix.

In the TMP_InputField.cs file around line 3134, make the following changes.

// Replace the line below
if (m_RichText)
    fullText = text.Substring(0, m_StringPosition) +  "<u>" + compositionString + "</u>" + text.Substring(m_StringPosition);
else
    fullText = text.Substring(0, m_StringPosition) +  compositionString + text.Substring(m_StringPosition);

// By this single line
fullText = text.Substring(0, m_StringPosition) +  compositionString + text.Substring(m_StringPosition);

In terms of the permanent fix, I need to re-work a big chunk of the input field to isolate the user input from the internal text used for display. This would enable the developer to still use rich text for coloring, etc. while being able to disable rich text from the user’s point of view.

HI, [mention|e2/s7rgWYU/1mMbd71EzRA==],my issue same as #11 ,i can’t input any Chinese with IME in unity editor and win standalone.
I use the new input system(1.0.0-preview 6), is this a reason for the issue?

That could be an issue and make sure you are using the latest release of the TMP package which is Preview 8.

1 Like

@Stephan_B This bug is breaking, and seems to still exist (unless it was fixed really recently - last month):

  1. Sometimes Chinese IME (BoPoMoFo / sometimes called ZhuYin) will replace the first character with an English one.
  2. If you finish a sentence, but didn’t press , the Chinese sentence will automatic type twice.
  3. After #2 when the input dupes, you need to use ctrl+space to change back to Chinese (seems like it throws them in English IME?).

I’ll have a new video soon. You can also ask tons of people that experience it directly here: 謊言王座 👑

You may @anon_79495420 or possibly direct message DoubleHeat_TW , or ask in #general (many folks speak English and are dying to get this fixed, so they’ll help you) while waiting for a response.

I did fix most IME issues in 2018.4. I still have to get those fixed in 2019.x or newer which I didn’t do because of the New Input System.

So please check if the behavior is correct in 2018.4. If it is then I’ll try to get this fixed once and for all for the newer versions of Unity.

Ah I cannot currently test earlier versions of Unity - I only have the time for smaller tests. With the new release of Chinese support, I’m currently flooded with work. Is it possible to snag a Unity QA for this one?

However, when we first discussed this in 2019, I was using an earlier version of Unity. At one point, I stopped responding here because I thought it was fixed. MAYBE it really was fixed in an earlier version before I upgraded (I’m now on 2020.1).

If you have any smaller tests for me todo to speed things along, give me a shout.

Let me get these fixes in 2019.x and 2020.x. I’ll try to get this done over the next few days and provide an update once it is done.

2 Likes

Video @ https://cdn.discordapp.com/attachments/756459120743284776/758586590791729183/zh_1st-key_bug.mp4

Video @ https://cdn.discordapp.com/attachments/756459120743284776/758587609961988096/zh_duplicated_bug.mp4

Let me know if you get delayed - not to add pressure, but I’m going to wait on this update before our own update. I think this is the top hurting point for Unicode right now. Let me know if you have any questions and I’ll try to ask the community that uses this.

Looking forward to the future fixes, it’s been almost two years since I filed a bug report to this bug. Hope that it will get fixed before I celebrate Happy Birthday to my bug report.

1 Like