How do you make a custom handle respond to the mouse?

a436t4ataf · May 25, 2013, 8:49pm

Handles are hugely important but apparently undocumented. Does anyone know the CORRECT way to make a custom Handle that responds to mouse-clicks/drags - or have a link to actual docs? (the Unity docs are blank or wrong in each case)

Using the built-in Handles is easy (and documented). But I want handles that e.g. let me connect (in my custom data) two GameObjects together by dragging a line between them. I know Unity can do this, but my ways of doing it so far are ugly hacks :(.

FYI For anyone else trying to do this, here’s some of the missing docs I’ve worked out by trial and error. This only works if you want to disable all controls and only use YOUR control (which is terrible, but it works) :

implement OnSceneGUI
1. first line you must “create” a controlID using the incorrectly named method: GUIUtility.GetControlID()
2. also you must read the value of Event.current.type
  1. if its “layout”, you must call “Handles.AddDefaultControl( id )”
  2. (you have to do this every frame: re-create your ID)
3. if its not “layout” then you can read the mouse position and button state etc from Event.current
Congratulations, you’ve broken Unity’s selection system, and nothing will work except your custom Handle

The docs talk about a NON default handle - AddHandle - which implies they have a system for selecting Handles by automatically finding the nearest Handle.

However, they have deleted half the docs for this, all that’s left is a reference to a non-existent attribute “nearestControl” (which is still documented on some other sites - e.g. unity.ru (the Russian version of Unity.com ???))

Most Unity packages ignore the above and use various hacks that “sort-of” do the same thing, e.g.:

(all of the working examples I found start by disabling everything, so you can’t use any built-in handles any more)
Pick a random number and use it as the ID
Always use 0 as the ID
Use any number you want for ID - so long as its negative (!)
Re-implement the whole logic of Unity’s Editor to simulate re-enabling the other controls

…and beyond that, nothing seems to work.

higekun · January 16, 2014, 1:06pm

I spent ages looking for a fully worked example of how to ‘properly’ implement custom handles. In the end I had to decompile UnityEditor (I recommend JetBrains dotPeek) and look inside Handles.FreeMoveHandle to see how to do it.

In case it saves time for someone else, here is a complete example of a custom handle. It’s similar to FreeMoveHandle, minus all the vertex snapping stuff, plus it returns mouse events so you can detect clicks, drags etc per handle.

public class MyHandles
{
	// internal state for DragHandle()
	static int s_DragHandleHash = "DragHandleHash".GetHashCode();
	static Vector2 s_DragHandleMouseStart;
	static Vector2 s_DragHandleMouseCurrent;
	static Vector3 s_DragHandleWorldStart;
	static float s_DragHandleClickTime = 0;
	static int s_DragHandleClickID;
	static float s_DragHandleDoubleClickInterval = 0.5f;
	static bool s_DragHandleHasMoved;

	// externally accessible to get the ID of the most resently processed DragHandle
	public static int lastDragHandleID;

	public enum DragHandleResult
	{
		none = 0,

		LMBPress,
		LMBClick,
		LMBDoubleClick,
		LMBDrag,
		LMBRelease,

		RMBPress,
		RMBClick,
		RMBDoubleClick,
		RMBDrag,
		RMBRelease,
	};

	public static Vector3 DragHandle(Vector3 position, float handleSize, Handles.DrawCapFunction capFunc, Color colorSelected, out DragHandleResult result)
	{
		int id = GUIUtility.GetControlID(s_DragHandleHash, FocusType.Passive);
		lastDragHandleID = id;

		Vector3 screenPosition = Handles.matrix.MultiplyPoint(position);
		Matrix4x4 cachedMatrix = Handles.matrix;

		result = DragHandleResult.none;

		switch (Event.current.GetTypeForControl(id))
		{
		case EventType.MouseDown:
			if (HandleUtility.nearestControl == id && (Event.current.button == 0 || Event.current.button == 1))
			{
				GUIUtility.hotControl = id;
				s_DragHandleMouseCurrent = s_DragHandleMouseStart = Event.current.mousePosition;
				s_DragHandleWorldStart = position;
				s_DragHandleHasMoved = false;

				Event.current.Use();
				EditorGUIUtility.SetWantsMouseJumping(1);

				if (Event.current.button == 0)
					result = DragHandleResult.LMBPress;
				else if (Event.current.button == 1)
					result = DragHandleResult.RMBPress;
			}
			break;

		case EventType.MouseUp:
			if (GUIUtility.hotControl == id && (Event.current.button == 0 || Event.current.button == 1))
			{
				GUIUtility.hotControl = 0;
				Event.current.Use();
				EditorGUIUtility.SetWantsMouseJumping(0);

				if (Event.current.button == 0)
					result = DragHandleResult.LMBRelease;
				else if (Event.current.button == 1)
					result = DragHandleResult.RMBRelease;

				if (Event.current.mousePosition == s_DragHandleMouseStart)
				{
					bool doubleClick = (s_DragHandleClickID == id) &&
						(Time.realtimeSinceStartup - s_DragHandleClickTime < s_DragHandleDoubleClickInterval);

					s_DragHandleClickID = id;
					s_DragHandleClickTime = Time.realtimeSinceStartup;

					if (Event.current.button == 0)
						result = doubleClick ? DragHandleResult.LMBDoubleClick : DragHandleResult.LMBClick;
					else if (Event.current.button == 1)
						result = doubleClick ? DragHandleResult.RMBDoubleClick : DragHandleResult.RMBClick;
				}
			}
			break;

		case EventType.MouseDrag:
			if (GUIUtility.hotControl == id)
			{
				s_DragHandleMouseCurrent += new Vector2(Event.current.delta.x, -Event.current.delta.y);
				Vector3 position2 = Camera.current.WorldToScreenPoint(Handles.matrix.MultiplyPoint(s_DragHandleWorldStart))
					+ (Vector3)(s_DragHandleMouseCurrent - s_DragHandleMouseStart);
				position = Handles.matrix.inverse.MultiplyPoint(Camera.current.ScreenToWorldPoint(position2));

				if (Camera.current.transform.forward == Vector3.forward || Camera.current.transform.forward == -Vector3.forward)
					position.z = s_DragHandleWorldStart.z;
				if (Camera.current.transform.forward == Vector3.up || Camera.current.transform.forward == -Vector3.up)
					position.y = s_DragHandleWorldStart.y;
				if (Camera.current.transform.forward == Vector3.right || Camera.current.transform.forward == -Vector3.right)
					position.x = s_DragHandleWorldStart.x;

				if (Event.current.button == 0)
					result = DragHandleResult.LMBDrag;
				else if (Event.current.button == 1)
					result = DragHandleResult.RMBDrag;

				s_DragHandleHasMoved = true;

				GUI.changed = true;
				Event.current.Use();
			}
			break;

		case EventType.Repaint:
			Color currentColour = Handles.color;
			if (id == GUIUtility.hotControl && s_DragHandleHasMoved)
				Handles.color = colorSelected;

			Handles.matrix = Matrix4x4.identity;
			capFunc(id, screenPosition, Quaternion.identity, handleSize);
			Handles.matrix = cachedMatrix;

			Handles.color = currentColour;
			break;

		case EventType.Layout:
			Handles.matrix = Matrix4x4.identity;
			HandleUtility.AddControl(id, HandleUtility.DistanceToCircle(screenPosition, handleSize));
			Handles.matrix = cachedMatrix;
			break;
		}

		return position;
	}
}

The most important things to understand are (A) the use of HandleUtility.nearestControl and HandleUtility.hotControl to manage input focus, with IDs generated by GUIUtility.GetControlID() and (B) the way OnSceneGUI is called multiple times for different events requiring very different handling.

Use it like:

	void OnSceneGui()
	{
		MyHandles.DragHandleResult dhResult;
		Vector3 newPosition = MyHandles.DragHandle(position, size, Handles.SphereCap, Color.red, out dhResult);

		switch (dhResult)
		{
		case MyHandles.DragHandleResult.LMBDoubleClick:
			// do something
			break;
		}
	}

a436t4ataf · May 26, 2013, 12:31am

More trial and error, I worked out:

OnSceneGUI() acts as though it’s written entirely in GL immediate mode. Once you realise this, a lot of the weird (undocumented) behaviour makes a lot more sense.
(if you don’t know about building 3D GUI in GL immediate, google it - it’s an old technique, been around a long time)
EditorGUIUtility and GUIUtility do nothing in scene GUI - except for providing the GetControlID() method. All other methods and variables are broken on those classes
You’re supposed to create a separate ControlID for every individual handle in your scene
ControlID’s appear to be re-assigned to you in the same order every frame, so you can persist them across frames (this would make sense c.f. GL Immediate mode)
-1 is a magic ControlID. It means “none”.
Once you become Default control, you can never find out if you still are, or if you’ve had it taken away, or if you’re the active control – you have to guess from then until you surrender it (by setting Default control to none)

…with this, I managed to build a full Handles-based editor GUI that doesn’t conflict with the existing handles, yay.