Creating a terain editor tool and cannot get ONSceneGUI to run

Unity Engine
1

I will post my code below but I do not see a problem with this, not sure exactly what you would need to see so I will post the whole 500 lines, not sure where else to go to ask for assistance, I have done this successfully before but not as indepth and also it was years ago. so I am not sure where I am going wrong. OnSceneGUI is called towards the bottom, most this code is an editor GUI

If it is relavant I am attempting to cast a ray to detect a tile then raise it by a desired ammount, if anybody knows a more efficient way of going about doing this feel free to help with any ideas as well while I’m here.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEditor;

public class WorldCreationTool : EditorWindow
{
    //Define menu icons.
    public Texture flat;

    public Texture inclineEnabled;
    public Texture inclineDisabled;

    public Texture hillEnabled;
    public Texture hillDisabled;

    public Texture cliffEnabled;
    public Texture cliffDisabled;

    public Texture add;
    public Texture minus;

    //Define tile variants.
    public Texture currentIcon;

    //Define data
    public TileSets setData;
    public TileSet currentTileSet = new TileSet();

    //Define what tool to use.
    public bool flatTool = true;
    public bool inclineTool = false;
    public bool hillTool = false;
    public bool cliffTool = false;

    public bool raiseTool = false;
    public bool lowerTool = false;

    public int tileVariant = 1;

    [MenuItem("Envirnment Tools/World Editor Tool")]
    static void Init()
    {
        // Get existing open window or if none, make a new one:
        WorldCreationTool window = (WorldCreationTool)GetWindow(typeof(WorldCreationTool));

        window.maxSize = new Vector2(644f, 780f);
        window.minSize = window.maxSize;

        window.Show();
    }

    void OnGUI()
    {
        //Define label style//
        var labelStyle = GUI.skin.GetStyle("Label"); //Define label style.
        labelStyle.alignment = TextAnchor.MiddleCenter; //Define label position.
        labelStyle.fontSize = 46; //Define label size.

        GUILayout.Label("World Creation Tool", labelStyle, GUILayout.Width(Screen.width), GUILayout.Height(46)); //Main Title

        ///////////////////////////////////////////Current Tool///////////////////////////////////////////////////

        if (!currentTileSet.defaultTile) //Set the current tile set to the default tile set.
            currentTileSet = setData.defaultSet;

        if (!currentIcon) //Set the current icon to the default tile set.
            currentIcon = currentTileSet.defaultIcon;

        Texture flatIconCurrent = flat; //Configure the flat icon to use.

        Texture inclineIconCurrent = inclineDisabled; //Configure the incline icon to use.
        if (currentTileSet.hasIncline)
            inclineIconCurrent = inclineEnabled;

        Texture hillIconCurrent = hillDisabled; //Configure the hill icon to use.
        if (currentTileSet.hasHill)
            hillIconCurrent = hillEnabled;

        Texture CliffIconCurrent = cliffDisabled; //Configure the cliff icon to use.
        if (currentTileSet.hasCliff)
            CliffIconCurrent = cliffEnabled;

        //Background Properties
        var backgroundStyle = GUI.skin.GetStyle("Box");

        //Begin tool box//
        GUILayout.BeginArea(new Rect(8, 52, Screen.width - 16, 170), backgroundStyle);
        GUILayout.BeginHorizontal();
        GUILayout.BeginVertical();
        GUILayout.BeginHorizontal();

        labelStyle.fontSize = 18; //Define label size.
        GUILayout.Label(currentTileSet.setName, labelStyle, GUILayout.Width(120), GUILayout.Height(32));

        labelStyle.fontSize = 24; //Define label size.
        labelStyle.alignment = TextAnchor.UpperCenter; //Define Label position.
        GUILayout.Label("Current Brush", labelStyle, GUILayout.Width((Screen.width / 2) + 16), GUILayout.Height(32));

        GUILayout.EndHorizontal();

        //Define current terrain icon.
        GUILayout.Box(currentIcon, GUILayout.Width(120), GUILayout.Height(120));

        GUILayout.EndVertical();
        GUILayout.BeginVertical();

        labelStyle.alignment = TextAnchor.MiddleLeft; //Define Label position.
        labelStyle.fontSize = 12; //Define label size.

        if (flatTool)
        {
            GUILayout.Label("Flat Tool", labelStyle, GUILayout.Width(80), GUILayout.Height(16));
            GUILayout.Box(flatIconCurrent, GUILayout.Width(40), GUILayout.Height(40));
        }
        if (inclineTool)
        {
            GUILayout.Label("Incline Tool", labelStyle, GUILayout.Width(80), GUILayout.Height(16));
            GUILayout.Box(inclineIconCurrent, GUILayout.Width(40), GUILayout.Height(40));
        }
        if (hillTool)
        {
            GUILayout.Label("Hill Tool", labelStyle, GUILayout.Width(80), GUILayout.Height(16));
            GUILayout.Box(hillIconCurrent, GUILayout.Width(40), GUILayout.Height(40));
        }
        if (cliffTool)
        {
            GUILayout.Label("Cliff Tool", labelStyle, GUILayout.Width(80), GUILayout.Height(16));
            GUILayout.Box(CliffIconCurrent, GUILayout.Width(40), GUILayout.Height(40));
        }

        //Define if raise or lower tool is in use
        if (raiseTool)
        {
            GUILayout.Label("Raise Tool", labelStyle, GUILayout.Width(80), GUILayout.Height(16));
            GUILayout.Box(add, GUILayout.Width(40), GUILayout.Height(40));
        }
        if (lowerTool)
        {
            GUILayout.Label("Lower Tool", labelStyle, GUILayout.Width(80), GUILayout.Height(16));
            GUILayout.Box(minus, GUILayout.Width(40), GUILayout.Height(40));
        }

        GUILayout.EndVertical();
        GUILayout.EndHorizontal();
        GUILayout.EndArea();
        ///////////////////////////////////////////Current Tool///////////////////////////////////////////////////


        ///////////////////////////////////////////Terrain Tool///////////////////////////////////////////////////

        //Background Properties
        GUILayout.BeginArea(new Rect(8, 230, Screen.width - 16, 160), backgroundStyle); //Define container

        //Define title properties.
        labelStyle.fontSize = 18; //Define label size.
        GUILayout.Label("Terrain Brushes", labelStyle, GUILayout.Width(Screen.width), GUILayout.Height(32)); //Title

        GUILayout.BeginHorizontal();
        GUILayout.BeginVertical();

        //Define tile set label properties.
        labelStyle.fontSize = GUI.skin.font.fontSize; //Define label size.
        GUILayout.Label(setData.defaultSet.setName, labelStyle, GUILayout.Width(80), GUILayout.Height(32));

        //Define terrain button properties.
        var buttonStyle = GUI.skin.GetStyle("Button");

        if (GUILayout.Button(setData.defaultSet.defaultIcon, buttonStyle, GUILayout.Width(80), GUILayout.Height(80))) //Button
        {
            currentTileSet = setData.defaultSet;
            currentIcon = currentTileSet.defaultIcon;
            tileVariant = 1;
        }
        GUILayout.EndVertical();

        for (int x = 0; x < 4; x++)
            if (setData.additionalSets.Count >= x)
            {
                GUILayout.BeginVertical();
                GUILayout.Label(setData.additionalSets[x].setName, labelStyle, GUILayout.Width(80), GUILayout.Height(32));
                if (GUILayout.Button(setData.additionalSets[x].defaultIcon, buttonStyle, GUILayout.Width(80), GUILayout.Height(80))) //Button
                {
                    currentTileSet = setData.additionalSets[x];
                    currentIcon = currentTileSet.defaultIcon;
                    tileVariant = 1;

                    //All the default sets that do not have cliffs don't have hills either so this works as of now.
                    //This method may prove troublesome later if made to be modular.
                    if (!setData.additionalSets[x].hasHill)
                    {
                        if (hillTool || cliffTool) {
                            flatTool = true;
                            inclineTool = false;
                            hillTool = false;
                            cliffTool = false;

                            raiseTool = false;
                            lowerTool = false;
                        }
                    }
                }
                GUILayout.EndVertical();
            }
        GUILayout.EndHorizontal();
        GUILayout.EndArea();
        ///////////////////////////////////////////Terrain Tool///////////////////////////////////////////////////


        ///////////////////////////////////////////Tile Variant///////////////////////////////////////////////////

        backgroundStyle = GUI.skin.GetStyle("Box");
        GUILayout.BeginArea(new Rect(8, 398, Screen.width - 16, 114), backgroundStyle); //Define container

        labelStyle.fontSize = 18; //Define label size.
        GUILayout.Label("Tile Variants", labelStyle, GUILayout.Width(Screen.width), GUILayout.Height(32)); //Title

        GUILayout.BeginHorizontal();

        if (GUILayout.Button(currentTileSet.defaultIcon, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
        {
            tileVariant = 1;
            currentIcon = currentTileSet.defaultIcon;
        }

        if (currentTileSet.iconTwo)
            if (GUILayout.Button(currentTileSet.iconTwo, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 2;
                currentIcon = currentTileSet.iconTwo;
            }

        if (currentTileSet.iconThree)
            if (GUILayout.Button(currentTileSet.iconThree, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 3;
                currentIcon = currentTileSet.iconThree;
            }

        if (currentTileSet.iconFour)
            if (GUILayout.Button(currentTileSet.iconFour, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 4;
                currentIcon = currentTileSet.iconFour;
            }

        if (currentTileSet.iconFive)
            if (GUILayout.Button(currentTileSet.iconFive, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 5;
                currentIcon = currentTileSet.iconFive;
            }

        if (currentTileSet.iconSix)
            if (GUILayout.Button(currentTileSet.iconSix, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 6;
                currentIcon = currentTileSet.iconSix;
            }

        if (currentTileSet.iconSeven)
            if (GUILayout.Button(currentTileSet.iconSeven, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 7;
                currentIcon = currentTileSet.iconSeven;
            }

        if (currentTileSet.iconEight)
            if (GUILayout.Button(currentTileSet.iconEight, buttonStyle, GUILayout.Width(75), GUILayout.Height(75))) //Button
            {
                tileVariant = 8;
                currentIcon = currentTileSet.iconEight;
            }

        GUILayout.EndHorizontal();
        GUILayout.EndArea();

        ///////////////////////////////////////////Tile Variant///////////////////////////////////////////////////


        ////////////////////////////////////////////Terrain Tool//////////////////////////////////////////////////

        Texture flatIconToUse = flat; //Configure the incline icon to use.

        Texture inclineIconToUse = inclineDisabled; //Configure the incline icon to use.
        if (currentTileSet.hasIncline)
            inclineIconToUse = inclineEnabled;

        Texture hillIconToUse = hillDisabled; //Configure the hill icon to use.
        if (currentTileSet.hasHill)
            hillIconToUse = hillEnabled;

        Texture CliffIconToUse = cliffDisabled; //Configure the cliff icon to use.
        if (currentTileSet.hasCliff)
            CliffIconToUse = cliffEnabled;

        GUILayout.BeginArea(new Rect(8, 520, Screen.width - 16, 260), backgroundStyle); //Define container

        GUILayout.Label("Terrain Tools", labelStyle, GUILayout.Width(Screen.width), GUILayout.Height(32)); //Title

        GUILayout.BeginHorizontal();

        //Define the style for the tool labels.
        labelStyle.fontSize = GUI.skin.font.fontSize; //Define label size.

        var toolButtons = GUI.skin.GetStyle("Button");

        GUILayout.BeginVertical();

        GUILayout.Label("Flat", labelStyle, GUILayout.Width(60), GUILayout.Height(32));

        if (GUILayout.Button(flatIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80))) //Button
        {
            flatTool = true;
            inclineTool = false;
            hillTool = false;
            cliffTool = false;

            raiseTool = false;
            lowerTool = false;
        }
        GUILayout.EndVertical();

        if (currentTileSet.hasIncline)
        {
            GUILayout.BeginVertical();

            GUILayout.Label("Incline", labelStyle, GUILayout.Width(60), GUILayout.Height(32));

            if (GUILayout.Button(inclineIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80))) //Button
            {
                flatTool = false;
                inclineTool = true;
                hillTool = false;
                cliffTool = false;

                //Cannot use a height altering tool without determining which direction to alter the height.
                //This method may prove troublesome later if made to be modular.
                if (!raiseTool && !lowerTool)
                    raiseTool = true;
            }

            if (!flatTool)
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Raise Tool", labelStyle, GUILayout.Width(90), GUILayout.Height(32));
                if (GUILayout.Button(add, GUILayout.Width(60), GUILayout.Height(60))) //Blank
                {
                    raiseTool = true;
                    lowerTool = false;
                }
                GUILayout.EndVertical();
            }
            else
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Raise Tool", GUILayout.Width(90), GUILayout.Height(32));
                GUILayout.Box(add, toolButtons, GUILayout.Width(60), GUILayout.Height(60)); //Blank
                GUILayout.EndVertical();

            }

            GUILayout.EndVertical();
        }
        else
        {
            GUILayout.BeginVertical();
            GUILayout.Label("Incline", labelStyle, GUILayout.Width(60), GUILayout.Height(32));
            GUILayout.Box(inclineIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80)); //Blank

            if (!flatTool)
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Raise Tool", labelStyle, GUILayout.Width(90), GUILayout.Height(32));
                if (GUILayout.Button(add, GUILayout.Width(60), GUILayout.Height(60))) //Blank
                {
                    raiseTool = true;
                    lowerTool = false;
                }
                GUILayout.EndVertical();
            }
            else
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Raise Tool", GUILayout.Width(90), GUILayout.Height(32));
                GUILayout.Box(add, toolButtons, GUILayout.Width(60), GUILayout.Height(60)); //Blank
                GUILayout.EndVertical();

            }

            GUILayout.EndVertical();
        }

        if (currentTileSet.hasHill)
        {
            GUILayout.BeginVertical();

            GUILayout.Label("Hill", labelStyle, GUILayout.Width(60), GUILayout.Height(32));

            if (GUILayout.Button(hillIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80))) //Button
            {
                flatTool = false;
                inclineTool = false;
                hillTool = true;
                cliffTool = false;

                //Cannot use a height altering tool without determining which direction to alter the height.
                //This method may prove troublesome later if made to be modular.
                if (!raiseTool && !lowerTool)
                    raiseTool = true;
            }

            if (!flatTool)
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Lower Tool", labelStyle, GUILayout.Width(90), GUILayout.Height(32));
                if (GUILayout.Button(minus, GUILayout.Width(60), GUILayout.Height(60))) //Blank
                {
                    raiseTool = false;
                    lowerTool = true;
                }
                GUILayout.EndVertical();
            }
            else
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Lower Tool", GUILayout.Width(90), GUILayout.Height(32));
                GUILayout.Box(minus, toolButtons, GUILayout.Width(60), GUILayout.Height(60)); //Blank
                GUILayout.EndVertical();
            }

            GUILayout.EndVertical();
        }
        else
        {
            GUILayout.BeginVertical();
            GUILayout.Label("Hill Tool", labelStyle, GUILayout.Width(70), GUILayout.Height(32));
            GUILayout.Box(hillIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80)); //Blank

            if (!flatTool)
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Lower Tool", labelStyle, GUILayout.Width(90), GUILayout.Height(32));
                if (GUILayout.Button(minus, GUILayout.Width(60), GUILayout.Height(60))) //Blank
                {
                    raiseTool = false;
                    lowerTool = true;
                }
                GUILayout.EndVertical();
            }
            else
            {
                GUILayout.BeginVertical();
                GUILayout.Label("Lower Tool", GUILayout.Width(90), GUILayout.Height(32));
                GUILayout.Box(minus, toolButtons, GUILayout.Width(60), GUILayout.Height(60)); //Blank
                GUILayout.EndVertical();
            }

            GUILayout.EndVertical();
        }

        if (currentTileSet.hasCliff)
        {
            GUILayout.BeginVertical();

            GUILayout.Label("Cliff Tool", labelStyle, GUILayout.Width(70), GUILayout.Height(32));

            if (GUILayout.Button(CliffIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80))) //Button
            {
                flatTool = false;
                inclineTool = false;
                hillTool = false;
                cliffTool = true;

                //Cannot use a height altering tool without determining which direction to alter the height.
                //This method may prove troublesome later if made to be modular.
                if (!raiseTool && !lowerTool)
                    raiseTool = true;
            }
            GUILayout.EndVertical();
        }
        else
        {
            GUILayout.BeginVertical();
            GUILayout.Label("Cliff Tool", labelStyle, GUILayout.Width(70), GUILayout.Height(32));
            GUILayout.Box(CliffIconToUse, toolButtons, GUILayout.Width(80), GUILayout.Height(80)); //Blank
            GUILayout.EndVertical();
        }

        GUILayout.EndHorizontal();
        GUILayout.EndArea();
    }

    //////////////////////////////////////////////Terrain Tool/////////////////////////////////////////////////////


    //////////////////////////////////////////////////Cast Ray////////////////////////////////////////////////////

    [CustomEditor(typeof(Tile))]
    public void OnSceneGUI()
    {
        Debug.Log("Firing Ray");

        Ray ray = HandleUtility.GUIPointToWorldRay(Event.current.mousePosition);
        RaycastHit hit;

        if (Physics.Raycast(ray, out hit))
        {
            if (hit.transform.tag == "Tile")
            {

                if (flatTool)
                    FlatTool();

                else if (inclineTool)
                    InclineTool();

                else if (hillTool)
                    HillTool();

                else if (cliffTool)
                    CliffTool();

                Selection.objects = new Object[0];
            }
        }
    }

    //////////////////////////////////////////////////Cast Ray////////////////////////////////////////////////////


    ////////////////////////////////////////////////Flat Tool/////////////////////////////////////////////////////

    void FlatTool()
    {
        Debug.Log("Flat Tool");
    }

    ////////////////////////////////////////////////Flat Tool/////////////////////////////////////////////////////


    //////////////////////////////////////////////Incline Tool////////////////////////////////////////////////////

    void InclineTool()
    {
        if (raiseTool)
            Debug.Log("Incline raise Tool");
        else if (raiseTool)
            Debug.Log("Incline lower Tool");
    }

    //////////////////////////////////////////////Incline Tool////////////////////////////////////////////////////


    ////////////////////////////////////////////////Hill Tool/////////////////////////////////////////////////////

    void HillTool()
    {
        if (raiseTool)
            Debug.Log("Hill raise Tool");
        else if (raiseTool)
            Debug.Log("Hill lower Tool");
    }

    ////////////////////////////////////////////////Hill Tool/////////////////////////////////////////////////////


    ////////////////////////////////////////////////Cliff Tool////////////////////////////////////////////////////

    void CliffTool()
    {
        if (raiseTool)
            Debug.Log("Cliff raise Tool");
        else if (raiseTool)
            Debug.Log("Cliff lower Tool");
    }

    ////////////////////////////////////////////////Cliff Tool////////////////////////////////////////////////////
}
2

Excellent. Go make an ULTRA simple cheeseball editor window or editor script that uses the OnSceneGUI() editor to do something trivial with some dummy class, and get that fully working.

Then reason about how that super-short simple script differs from the massive thing above.

What is often happening in these cases is one of the following:

  • the code you think is executing is not actually executing at all
  • the code is executing far EARLIER or LATER than you think
  • the code is executing far LESS OFTEN than you think
  • the code is executing far MORE OFTEN than you think
  • the code is executing on another GameObject than you think it is
  • you’re getting an error or warning and you haven’t noticed it in the console window

To help gain more insight into your problem, I recommend liberally sprinkling Debug.Log() statements through your code to display information in realtime.

Doing this should help you answer these types of questions:

  • is this code even running? which parts are running? how often does it run? what order does it run in?
  • what are the values of the variables involved? Are they initialized? Are the values reasonable?
  • are you meeting ALL the requirements to receive callbacks such as triggers / colliders (review the documentation)

Knowing this information will help you reason about the behavior you are seeing.

If your problem would benefit from in-scene or in-game visualization, Debug.DrawRay() or Debug.DrawLine() can help you visualize things like rays (used in raycasting) or distances.

You can also call Debug.Break() to pause the Editor when certain interesting pieces of code run, and then study the scene manually, looking for all the parts, where they are, what scripts are on them, etc.

You can also call GameObject.CreatePrimitive() to emplace debug-marker-ish objects in the scene at runtime.

You could also just display various important quantities in UI Text elements to watch them change as you play the game.

If you are running a mobile device you can also view the console output. Google for how on your particular mobile target, such as this answer or iOS: https://discussions.unity.com/t/700551 or this answer for Android: https://discussions.unity.com/t/699654

Another useful approach is to temporarily strip out everything besides what is necessary to prove your issue. This can simplify and isolate compounding effects of other items in your scene or prefab.

Here’s an example of putting in a laser-focused Debug.Log() and how that can save you a TON of time wallowing around speculating what might be going wrong:

https://discussions.unity.com/t/839300/3

You must find a way to get the information you need in order to reason about what the problem is.

3

I got it working, I needed a new class and to name it TileEditor. This presents a few problems however it only fires the ray and runs the other scripts with the debug log firing before the ray. Without it I cannot get it to fire the ray.

Edit: its working now that i zoomed in however that still makes no sense to me how the debug linbe could increase the distance of the ray. But it is working so I will take it.