With the rendering pipelines, tools like PolyBrush are now depreciated. How can I do this?

[A_World_Maker]

Hi,

Since the ilks of Polymesh are now depreciated and the tools to Vertex Paint within Unity Editor seem hidden, I am lost to where to turn.

I want to apply a moss texture to the base of my trees using a height map, or other control, to blend to the terrain my trees and plant meshes. I was using a shader that used Transmission, but it is not Rendering Pipeline compliant, or one that I can sell with my Foliage Packs.

I was trying to use a Blend Material, with two textures sets (Diffuse, Normal and AO) for each, but my skills in creating such materials are limited to following a tutorial. The only ones I can find on YT are all pre- Rendering Pipelines and years out of date.

The Unity Manuals on the subject were written when slate and chalk were fashionable so of little use. Does anyone know where I can find a YT tutorial that I can follow to create such a material/shader that are URP compatible?

The pictures show what I want to achieve, using a Material that I can include with my trees/foliage packs on the Asset Store, which excludes any Product on the Store due to the EULA, even if they are free.

[A_World_Maker]

I have managed to create a shader in Unreal, where there is Mesh paint tools built into the Editors, that covers all what I need, without infringing any copyright, as I created the materials myself. I can apply both colours and textures directly to the mesh using a vertex brush. The same way I could using Polybrush before the Rendering Pipelines.

Searching the web, not a single tutorial of how to achieve the same thing in Unity using the now De Facto built-in URP rendering pipeline. Most information relates to Polybrush, but with the introduction of URP, that is now defunct, with no replacement. I feel Unity has run off creating editors and pipelines, leaving a big gap behind with its Editors Toolset.

I, like so many Sellers on Unity cannot use a asset from the store, whether it is paid for or free due to the restrictions, EULA’s and Licences.

No mention from Unity about how they are going to fill this gap/polybrush that works with their wonderful new pipelines. I could find no information on Unitys own Docs, no information online, other than terrain materials, which are clearly not suitable.

[kdgalla]

I would use Blender for vertex painting.

@A_World_Maker

No mention from Unity about how they are going to fill this gap/polybrush that works with their wonderful new pipelines.

Don’t hold your breath…

[A_World_Maker]

Hi,

I have only used blender once for vertex painting. It was a tree and I found it cumbersome to use to be honest.

I have tried researching Materials that I could use for the process. All I have found so far, is a couple of videos that are 4-5 years old, they don’t speak specifically about the URP pipeline, so hard to follow. Unitys own documentation is several years out of date and the Materials section has no information about their pipeline shaders.

I do like the Pipelines as a progression, but despite its new pipelines, Unity 6 offers me very little above stability. Many useful tools have vanished and Unity do not respond when I enquired about some of them. At best I get, it’s on the roadmap…

I have even explored dropping the meshes into Unreal, as I normally do as part of my workflow… baking the materials into the mesh vertexes, and exporting the models as FBX and see if it translates into Unity. That is an awful lot of work for a relatively simple and default procedure of a standard editor/engine. Currently, of all the Editors/Engines I use, Unity is the only one that has no Model Vertex Painting tools.

[A_World_Maker]

Hi again, now I have digested your reply in greater detail, how would I assign a texture to the vertex painting in Unity? Would I have to do that within Blender, creating the UV’s to cope with that?

I have to be honest, I use many VR Editors, Graphics Programmes and Platforms in the 28 years I have been in VR… but Blender is like a ball of spaghetti to me, I cannot edit an existing Material as it requires me to change modes to gain a function that shows a material, but no texture slots, and normals are a black art… I have tried reading up on it, following videos, tutorials and even realtime with someone I know that uses it daily… but none of it makes sense or even has a logical flow.

I have Used AutoCad to design buildings, infrastructure, roadways, make 3D representations, then onto self taught VR Platforms like Atmosphere, Second Life, Unity, Unreal and many more VR Editors… None of which share any similarities with Blender. The shader has a node for a Normal connector, but you cannot simply attach a Normal Texture object to it… Even Blender For Artists doesn’t work for me.

I described Blender once, as being a cobbled together version of many separately evolved software modules, where there was no steering committee or Baseline Project Management rules. Then someone else created a menu system to cobble them all together. I have written/programmed Stock Control, Project Management and Shipping programmes all from scratch and all working flawlessly from a common menu system, that combine that with an Estimating Programme with Spons format BOQ output, yet Blender remains a mystery.

In Unreal, which I have been using for around 5-6 years, I can create complex Materials to do almost anything, including BluePrints for some automation. In Unity, until the introduction of the Pipelines, I was creating Materials to much the same, using its shaders and scripts… I can still apply Vertexting painting to my meshes in earlier versions of Unity, but no Materials exist anymore for Painting Vertexes with Textures, since the demise of Polybrush. Unity have failed its users not implementing new Pipelines without the Tools that work with it.

I am going to tryout applying Vertex Texturing to my meshes in Unreal, then exporting them as FBX and seeing what gets transfered… a Sad state of affairs really.

For instance, I cannot recreate this in Unity anymore…

I 'painted Moss onto the tree mesh, then onto a Dome Ground mesh attached to the tree, to give this localised effect. The Floor Moss is not on the landscape but on a mesh that is scalable, to create a bespoke and editable effect in unison with the Tree. All without using any special tools or techniques… I used to be able to do the same before the RP’s of Unity.

[kdgalla]

@A_World_Maker

Hi again, now I have digested your reply in greater detail, how would I assign a texture to the vertex painting in Unity?

Are you talking about something like a splatmap where you have multiple textures and you are using the vertex colors to determine how much of each texture to paint on a specific vertex? You’d just need the right shader. Sorry, when I posted earlier, I’d assumed that you already had the material that you needed and were just looking for a way to get the vertex colors. I tried to find something like that all ready to go, but this is the best I could find:

Blend Between Textures in a Shader (Vertex Colors) | ShaderGraph Unity Tutorial

Would I have to do that within Blender, creating the UV’s to cope with that?

There are two common ways to apply textures. One is vertex UVs (that you’d create in Blender) and the other is to use Triplanar projection mapping (usually world space).

Vertex UVs:
You should only need one set of UVs regardless of how many textures you use. They’ll all just be applied with the same UVs. If you have trees and stuff from the asset store, you can get by with the vertex UVs that are already there. You don’t need to do anything special for this type of material. If you’re making assets from scratch, you’d need to do a regular UV unwrap like you would for any textured mesh.

Triplanar is easier because then you don’t have to UV unwrap every single model, but the downside is that you have no precise control over which part of the texture goes over which part of the model. If you’re using generic bark and moss textures, then it should work, though. It’s pretty common to use world-space triplanar projection on terrain assets.

Edit: By the way, I suggested Blender in my previous post, but you can use any 3D software you want. My point was simply that Unity itself is not very useful for creating assets. It has some facilities, such as a terrain editor, probuilder, etc, but these are all minimal, janky, and not well maintained, as you’ve noticed. It’s better to think of Unity as just a game engine.

That being said, Materials are generally non-transferable because Unity, Unreal, Blender, and whatever, each have their own renderers and shaders and what not.

[A_World_Maker]

Hi,
Thank you for the response.

My current Workflow is to produce the assets, trees, plants, props etc in a series of different 3D modelling programmes, output those as OBJ. Sometimes I will convert those the FBX. In the same Mesh folder are stored all the Texture Maps, including an array of Maps for different Editors.

Opening up an Engine, I import those common Meshes, along with its MTL files, and Textures, bringing in additional maps to suit the Engines shaders.

At this point, none of the Meshes are Vertex Painted, except for the Wind Colourations held in the channel of the UVs.

These are then setup with Materials I create within each Engine/Version. Adding Vertex Painting and say Moss, Snow, Dead Leaves etc, is a new portion of the Workflow, to add to the products value to our clients.

This has worked across all the Engines I produce Assets for from the pool of meshes/textures I create. Unreal handles the process slightly different to Unity and other Engines, but they all share one thing in common, a ‘Toolset’ to do that work. Unity, since its Introduction of the Pipelines, has made the Polybrush tool defunct and without a replacement. I can see nothing on the roadmap to suggest a replacement is in the pipeline.

As you can see from the picture, I have created a workflow that uses a scalable ‘dome’ mesh that uses Nanites in Unreal, to give the Moss depth, that will serve well with Snow, Dried Leaves etc etc… that doesn’t need to use the Landscape Layer, with its washed out colours that do not match, and give greater flexibility. That can use features like Nanites and Tessellation, combined with Vertex Painting for greater control. Simply spraying a colour or texture doesn’t give control over Normals/Displacement etc. The Latter, Unity is way way behind other Engines on, now they no longer have Vertex Painting for the Engine, except in the landscape. Many techniques I have learnt, using Vertex colouring, are now no longer viable in Unity.

The addition of the Vertex Painting feature is relatively new for my Assets. I normally build a workflow from an early Engine version and progress that through the various Engine Versions for compatibility. No problem in all the other Engines I sell Assets for, but Unity has created a barrier with the dropping of Vertex painting tools compatible with the new rendering pipelines.

Here is the next issue to compound that.
Assuming I had a shader that would allow for texture assignment to vertex colour mapping, it cannot be one sourced from the asset store, as it would be included within my Packs for sale, breaking the EULA agreements. The same is true to other engines, but I do not need to buy anything, I can make them within my workflow for that Engine, except Unity since the Rendering Pipelines made the mesh vertex painting defunct as an option. Making Materials beyond the standard ones provided with the Engine, just for a single Engine, would add to the cost of the products. This is not the case in other Engines with Vertex Painting as standard.

For now, I will continue with the other Engines, with this additional Workflow and just not provide it with Unity Engine submissions in Editors that use URP/HDRP pipelines. Makes it even harder for submissions, showing screenshots of Products in Engine Versions, Pre/Post Pipelines, where features in the Pre versions are better than the Post versions. Unity 6 now has URP built in, which has improved several things, but has also taken a step backwards in areas in my opinion. What has made that worse, none of the Tutorials and Information have been updated for years in their manuals and website.

[A_World_Maker]

I am considering importing my ‘Holographic Plants’ into Unity, but I need to use the Editor version of the Vertex Painting.
The goal is to not colour the mesh vertices for the material, but for the gradient colouring, so the meshes are not monochrome.

The Material itself is doable for all versions of Unity, just not the Vertice Painting, for URP or other pipelines.

Anyone have any suggestions?

[Gandasdof]

I have lately been working on a simmilar tool, just instead of vertex painting face material assignment. You could build a simmilar tool for doing this. Here is an adapted version:

Regular Monobehaviour on the model you want to paint:

using UnityEngine;

[RequireComponent(typeof(MeshFilter))]
[RequireComponent(typeof(MeshRenderer))]
public class VertexPainter : MonoBehaviour
{
    public MeshFilter meshFilter;
    public bool paint = false;
    public Color paintColor = Color.white;
    public float paintRadius = 0.25f;
}

On a separate script have this Editor Script:

//put this file into an editor folder or assemby or add the #UNITY_EDITOR pragma to fix compiling

using System.Reflection;
using UnityEditor;
using UnityEngine;

[CustomEditor(typeof(VertexPainter))]
public class VertexPainterEditor : Editor
{
    VertexPainter t;
    private void OnEnable()
    {
        t = target as VertexPainter;
        if (t.meshFilter == null) t.meshFilter = t.GetComponent<MeshFilter>();//make sure we have a meshcomponent target
    }

    //This is an internal unity method for raycasting which isnt naturally exposed.
    //This is also the pramary reason why other tools have been depreciated.
    private static readonly MethodInfo intersectRayMeshMethod = typeof(HandleUtility).GetMethod("IntersectRayMesh", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static);

    private void OnSceneGUI()
    {
        //only accept valid events
        switch (Event.current.type)
        {
            case EventType.Layout:
            case EventType.Repaint:
            case EventType.MouseDown:
            case EventType.MouseUp:
            case EventType.MouseDrag:
                break;
            default: return;
        }


        if (!t.paint) return; //return if we have not painting enabled
        if (Event.current.alt) return; //return if alt is pressedto allow camera rotation

        //Get the hit point
        Ray ray = HandleUtility.GUIPointToWorldRay(Event.current.mousePosition); //Get the ray from mouse
        object[] rayMeshParameters = new object[] { ray, t.meshFilter.sharedMesh, t.transform.localToWorldMatrix, null }; //get the raycast parameters
        if ((bool)intersectRayMeshMethod.Invoke(null, rayMeshParameters)) // check if mouse is on top of the mesh
        {
            RaycastHit hit = (RaycastHit)rayMeshParameters[3];
            //possible uses
            //var UVs = hit.textureCoord;
            //Vector2 p0 = t.meshFilter.sharedMesh.vertices[t.meshFilter.sharedMesh.triangles[hit.triangleIndex * 3 + 0]];
            //Vector2 p1 = t.meshFilter.sharedMesh.vertices[t.meshFilter.sharedMesh.triangles[hit.triangleIndex * 3 + 1]];
            //Vector2 p2 = t.meshFilter.sharedMesh.vertices[t.meshFilter.sharedMesh.triangles[hit.triangleIndex * 3 + 2]];
            //Vector3 hitPoint = hit.point;

            //TODO Draw the gizmo
            Handles.DrawWireDisc(hit.point, hit.normal, t.paintRadius);
            for (int i = 0; i < t.meshFilter.sharedMesh.vertexCount; i++)
            {
                // Handles.DrawSolidDisc(t.transform.localToWorldMatrix.MultiplyPoint3x4(t.meshFilter.sharedMesh.vertices[i]), t.transform.rotation * t.meshFilter.sharedMesh.normals[i], 0.1f);

            }

            if (!((Event.current.type == EventType.MouseDown || Event.current.type == EventType.MouseDrag) && Event.current.button == 0)) return;//return if we are not actively painting

            //paint the vertex

            Color[] vertColors = (Color[])t.meshFilter.sharedMesh.colors.Clone();//clone the current vertex colors
            //all of this might not work if mesh does not have previous vetex colors so we fix it
            if (vertColors.Length != t.meshFilter.sharedMesh.vertexCount)
            {
                var newColors = new Color[t.meshFilter.sharedMesh.vertexCount];
                if (vertColors != null)
                {
                    for (int i = 0; i < vertColors.Length && i < newColors.Length; i++)
                    {
                        newColors[i] = vertColors[i];
                    }
                }
                vertColors = newColors;
            }
            //for each vertex we check distance to see if it is between paint distance
            for (int i = 0; i < t.meshFilter.sharedMesh.vertexCount; i++)
            {
                Vector3 worldVertex = t.transform.localToWorldMatrix.MultiplyPoint3x4(t.meshFilter.sharedMesh.vertices[i]);//convert vertex to world space
                var dist = Vector3.SqrMagnitude(worldVertex - hit.point);
                if (dist < (t.paintRadius * t.paintRadius))//Check if vertex is between paint distance (sqrt=faster)
                {
                    vertColors[i] = t.paintColor;
                }
            }
            t.meshFilter.sharedMesh.SetColors(vertColors);//set the colors back
            Event.current.Use(); //use the event to not misuse

        }

    }
}

I still dont know how to do editor undos correctly so i ommited them.

To blend the materials it would look something like this(in shadergraph) (i’ve only done it in red and green for albedo and normal)

I hope this works to fix your issue?
I dont remember who showed me the trick to get mesh raycasting but its a very needed trick which unity should expose by default, not with these weird workarounds.

[A_World_Maker]

Hello Hedenrag,

Thank you for taking the time to respond to my post. I have moved ahead with my efforts to do this in Unreal with great success, but not Unity.

I am afraid I am only an Artist and not a coder. What you provided looks well written but I have no clue what to do with it. Clearly your talents are within that realm.

Does it allow for me to paint the vertices of a mesh? Which Polybrush did perfectly.

In addition to using Polybrush to create other effects on plants etc, like Moss Growth and leaf piles, I was also considering Snow as another application… all can no longer be applied to meshes that use the new URP and HDRP.

I want to be able to use a tool within the Editor, to give greater scope for those that purchase my products, without having to create bespoke meshes for each variation, where materials alone will not give me the finished product I am looking for, that I can apply across many platforms, using Editor bound tools. Unity, has fallen way behind in that field.

[Gandasdof]

Yes, the tool i’ve provided does just that, it allows you to paint the vertices of a mesh object. To use it create two monovehaviours, One Named “VertexPainter” where you copy the first short code, the second monobehaviour should be named “VertexPainterEditor” and be placed in a folder named Editor (anywhere in your assets/package folder).
Now you can start using it, add the “VertexPainter” script to the object you want to paint and activate the paint button. It has prettty intuitive controls. I must say though that its not a very performant tool and is a bit laggy with high vertex coun,t but it should still prove usefull.

[A_World_Maker]

Hi again,
I will add these scripts and test it out on live models, grab some screenshots etc…

I am curious, I assume I have to leave the scripts in each model (VertexPainter), but the Editor script (VertexPainterEditor) remains on my system?
How will that work when I sell the meshes to others? Do I need to pass them the VertexPainterEditor script as well?
I assume, VertexPainterEditor is the Editor Tool, that enables the vertices to be painted, which is then stored in the mesh model as it would normally… and therefore not required by the end user…

Another question, will this work with all Rendering Pipelines?.. It is only Unity 6 Editor where I am forced to use URP and HDRP Rendering Pipelines, that breaks Polybrush. As far as I am aware (Artist not Coder) both the URP and HDRP do not have Materials that can be used on Meshes, other than Landscapes, that allow multiple Layers that is supported by Vertex painting.

Sorry if these queries sound obvious to you.

[Gandasdof]

I have actually only tested it in Unity 6000.0.37f1 altough i think i haven’t done anything version specific. The only script you need to have on the model is the VertexPainter script (the other script just references it) and when you are finished painting you can just remove it again and the painted vertices will remain.

I Also have found out that it currently did not save the changes on closing the editor see:Working with Model Prefabs and FBX Prefab Variants | FBX Exporter | 4.0.1. If you dont want to import the new package or wish to send your own version with your package you can also copy and paste the mesh part of the imported model.

I will be using vertex painting too so i had to fix it anyway.
I have packaged it all for easier update and install: GitHub - Hedenrag/com.doublegaugegames.vertexpainter: Allows for vertex painting a mesh inside Unity.

[A_World_Maker]

Hi,

I tried installing the package from the + manager using the Git Url function, but it came up an error. Any thoughts?

[Gandasdof]

Without any indication of the error i cannot help much. I think git may be a required dependency for that function to work? But a quick search of the error should show you how to fix it.

[A_World_Maker]

Ahh, I thought I had GIT on my system, quick check said otherwise.

The error only said in red text, cannot download from GIT server… the usual helpful ‘hints’…

[A_World_Maker]

Hi, Clearly having GIT on my system allowed me to install the package.

When I tried adding the Component Script to an existing mesh, I got some errors (red ones)… pasted below…

Asset Packages/com.doublegaugegames.vertexpainter/README.md has no meta file, but it’s in an immutable folder. The asset will be ignored.

Using the Save Mesh to File:
Couln’t create asset file!
UnityEdidtor.AssetDatabase:CreateAsset (UnityEngine.Object,string)
VertexPainterEditor:OnInspectorGUI () (at ./Library/PackageCache/com.doublegaugegames.vertexpainter@d74889e6c706/Editor/VertexPainterEditor.cs:69)
UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

UnityException: Creating asset at path Old oak Tree04.asset failed.
UnityEditor.AssetDatabase.CreateAsset (UnityEngine.Object asset, System.String path) (at <95707308f6f3498991e9df8902fe97ba>:0)
VertexPainterEditor.OnInspectorGUI () (at ./Library/PackageCache/com.doublegaugegames.vertexpainter@d74889e6c706/Editor/VertexPainterEditor.cs:69)
UnityEditor.UIElements.InspectorElement+<>c__DisplayClass79_0.b__0 () (at <95707308f6f3498991e9df8902fe97ba>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

I am using up to date Win10 OS and this editor Unity 6000.0.41f1 DX11.

On creating the mesh to file, then selecting that, the scale was reduced and I had to apply a 100 scale parameter. I also saw that the 2 materials had become used on all parts of the mesh (bark and leaves together).

[Gandasdof]

Asset Packages/com.doublegaugegames.vertexpainter/README.md has no meta file is my error but should not affect anything.
On top of the component there is this warning when editing a mesh file (fbx, obj, etc):

If this message shows you should click what currently says “create new mesh” which i will rename to “Create mesh copy” to avoid confusion. Save mesh at path should throw an error if you havent created a new mesh previously (It should state something like: Couldn't add object to asset file because the Mesh 'MeshName' is already an asset at 'Assets/PathToMesh/MeshFile.extension'!).

Save mesh at path isnt actualy necesary to save the changes made. If you make a copy but dont save to path the mesh gets saved in the prefab () as long as it is not a packaged prefab (with this symbol:).
Save to asset is only needed if you want to save the new mesh as a new asset either to assign it to other objects or to save variations.
There is no scaling in any part of my code so i dont think thats the issue but i know that from certain modeling programs you expert the file and when importing it to unity it autoscales them. I know from blender when I import the meshes they get scaled by 100% and rotated 90 degrees.
When saving the mesh you are only saving the mesh so any transformations/materials you added/changed are not saved with it. For doing that you need to save a Prefab with all changes.

[A_World_Maker]

I’m sorry, I cannot move past the point, where the two materials used on this model, are combined when applying the script component.

Any system must only add to a model, in a way I intend, not destroy the Alpha Channels, and combine two or more materials on the mesh. This can be seen clearly on the screen shot of before and after.

I checked both materials before and after, there were no changes, the leafs still showing their Alpha Channel, so there were no control over this happening.

I would note, that this model comes with LODs (LOD 0-4). Could these be the cause?

[Gandasdof]

This is hard to resolve without a simmilar model. I only tested simple models. Yours seems to have complex Information i might not have acounted for. Without a model to test this may be a dead end.