I am looking for a way to access (and modify) the generated mesh data of a VisualElement. In Unity 2021 and 2022 I was able to do that via accessing (using reflectinos) the “painter” field of type UIRStylePainter in the MeshGenerationContext (“m_Entries” then contained the mesh data as NativeSlices).
However, in Unity 2023 and Unity 6 that no longer exists and was replaced by a jobs based drawing mechanism. I fail to see where the final mesh data is aggregated (if there is such a place). The TempMeshAllocatorImpl seems to hold some vertex data in the pool but I don’t think that’s the way to go.
Yes, that is what I am subscribing (appending) to. I fail to see where the EXISTING mesh data is in the MeshGenerationContext. In 2022 I was able to get it from the mgc’s “painter.m_Entries” (nicely grouped by batches) but that field no longer exists in 2023 or 6.
My setup is that I have a Label. To that I then add a Manipulator. That manipulator then appends to generateVisualContent and tries to change the mesh before it is actually rendered.
I know that’s not a supported (public API) workflow. I am just looking for guidance on where to look in the UITK source.
What I have tracked down so far:
MeshGenerationContext.DrawText(…)
→ calls DrawTextBase()
→ calls DrawTextInfo()
→ calls MakeText()
→ this then creates entries via m_MeshGenerationContext.entryRecorder.DrawSdfText(…).
So I know the text generation stores its results as entries in the “EntryRecorder entryRecorder” of the MeshGenerationContext.
However, if I look at the data inside at runtime it’s all already deallocated:
I also tried changing vertex data via the uitkTextHandle of the Label but that did not work either. I assume that does not reference the actual vertex data but contains copies of it.
If I remember correctly, some of the tessellation work was moved to native C++ jobs instead of C#, which would explain why your setup doesn’t work anymore.
Trying to modify the mesh through internal APIs is going to be an uphill battle for you. Can you share more details about what kind of effect you’re trying to do? It would be nice for us to provide public ways to do this.
I am trying to make a generic text animation modifier. So I want a modifier that can be added to any TextElement that then takes the rendered mesh (vertices) and animates those. It works nicely in 2021 and 2022 but I have hit a wall with 2023 and Unity 6. If the actual mesh is out of reach of managed code then I guess my idea of using a modifier is dead. I have dug into the code and I found the Job that creates the vertices and it seems it communicates the vertices back:
but sadly those are cleared immediately afterwards leading to a dead end for me
The vertex information then goes to the UIMeshGenerator which allocates a TempMeshAllocatorImpl but sadly it then also immediately clears the vertices:
My next approach would be trying to modify the text vertices before the go into the drawing job. Is there any way I could change the text mesh vertices before they are handed off to tessellation? I have access to the TextElementInfos, LinkInfo etc. (also via reflection).
It would be nice to have some public facing API for changing UITK Meshes before rendering. That would allow for many effects. There already is this line
Hey! Indeed, we’ve multi-threaded the generation of the Text in U6. With this new optimization, I unfortunately don’t see a workaround for you.
I’m really sorry for the challenges you had to go through to make it work. Built-in support for text animation is something we need to provide. I can’t make any promises, but I’ll see if I can reprioritize it.
Thank you all. Did not expect three Unity employees to respond
For those interested. After digging into the UITKTextJobSystem I found a way but it’s rather cumbersome. The path to the vertex data that is used for rendering is under: TextElement.uitkTextHandle.textInfo.meshInfo[..].vertexData[...]
vertextData contains a list of TextCoreVertex structs that can be changed BEFORE they are rendered. Notice that it’s a struct so it’s a lot of code if you do change it via reflections. Also the changes are somewhat persistent (still have to figure out when that list is regenerated so I can apply my changes accordingly).
It’s not 100% what I wanted but it might be enough (still lot’s of things to test but I have a proof of concept running in Unity 6 ):
Reminder for myself: Look in the TextJobSystem for where the mesh is generated (MeshGenerator and Painter2D will sidetrack you). The whole entries thing seems to just be scaffoling for jobs and deferred execution.
Update: Turns out sadly this is not a reliable way. It works randomly about 50% of the time. I assume something threads related ( UITKTextJobSystem.Execute() calls visualElement.uitkTextHandle.UpdateMesh() which (if isDirty == true) regenerates the vertex data and kills my modifications).
Update 2:
Reminder for myself. The materialReferenceIndex seems to nicely match the meshInfos by index (batch). Remember to call UpdateMesh() BEFORE changing the vertices or else the TextCore will call it AFTER your changes and overwrites any changes. If kept in mind you can enjoy:
@ HugoBD-Unity : I ended up calling UpdateMesh() myself whenever needed (hash or isDirty). Then I change the vertex data before generateVisualContent is called. I hope that will keep working until we get a proper API
@HugoBD-Unity , @karl_jones
Final working result with manipulators and compatibility ranging from Unity 2022.3 to Unity 6:
I am pretty happy with it
A few remarks in terms of what would be helpful in future text access APIs:
A sorted list of TextInfos would be nice. Right now the layout is by batch so the order of vertices is “cumbersome” to determine because fonts and sprites will mess it up. In a potentially future API a nice ordered list of rendered quads with the corresponding charIndex in the original text (+ tag infos if part of tags) would be nice.
The tag parsing seems to have problems with nested tags like <link><link>text</link></link>. It only results in one LinkInfo object but it “should” be two. Also the link info seems to be updated late in the frame cycle. I had to add some waiting to make it work. To me this is relevant because I am piggy-backing on the link tag with a custom attribute, example: <link anim="something">Animated Text goes here</link>. I’ve also found that the (undocumented) link tag is about the only one that supports arbitrary attributes without showing them in the rendered text or breaking parsing. If possible it would be nice to have support for custom tags or at least have this behaviour made official so we can rely on <link> in the future.
In order to make the vertices available to me I have to call UpdateMesh() or Update() (depending on the Unity version). I am still not sure when exactly they are updated (especially in 2023+ since that’s multi-threaded). The forced mesh update does cause some memory allocations. It would be nice if a future API could provide an allocation free way of modifying the text vertices after all internal text generation is set and done.
Also right now I have to kinda regenerate the text to make the vertices update (waste of resources). This is the most brittle part of it as I am reaching into NativeSlices etc. to change the mesh directly.
Additional wishlist (things that would be nice to have):
Custom tags. At the moment using the link tag is fine but it would be nice if we could use custom tags in richt texts. All other tag infos are not available (or at least not in the TextInfo data structure). The current text parsing seems to be rather strict about what tags are supported. TMPro used to support custom tags enabling all sorts of nice functionality. Right now TextInfo only contains data about link tags.
It would be tremendously helpful if modifiers could add custom xml attributes to elements. This would enable an inheritance free workflow and allow us to write extensions that could leverage the UI Builder to configure manipulator details. As of now this has to be done elsewhere. Currently I am using Scriptable Objects for that and link them to elements via USS classes.
I am aware that this is asking a bit much (lot’s of internal APIs would need to be exposed) but as you can hopefully see above, the results are worth the effort.
Hi @_geo1! Thanks for sharing and for building such a great plugin. We’ve introduced a Text Animation API in 6000.2.0b9 that should make your life easier.
That API looks very promising. At first glance I did not see any tag infos in the glyph type but the vertex infos alone are already a huge improvement. Thank you
If I could wish for something more then it would be:
An easy way of matching tag infos to glyphs
Custom tag support
(optional) Custom shader support. Though I understand this would break batching, so users would have to use it with care.
Custom tags would make associating custom infos to text easier. An example would be a text that should show a shake animation for one word.
That could look like this: Hello to a <shake>strange</shake> world.
where only the word “strange” should show a shake animation.
Additionally it would be nice if Unity could parse the tags for us including attributes. That could look like this: Hello to a <delay duration=0.5 /><shake strength=5>strange</shake><delay duration=0.5 /> world.
where in addition to the shake animation tag it would also parse the “strength” attribute and the delay tags with their duration attribute (used for typewriter like animations).
Using that we would be able to associate tags and attributes with glyphs. It would open up a lot of possibilities.
Currently we can no do this easily because:
a) Custom tags are not parsed and shown as plain text.
b) The only tag that supports custom attributes (without breaking parsing) is the undocumented “link” tag.
):
