Test Automation: Simulate mouse down event at Vector2 location during runtime

Questions & Answers
1

I have a script that records the mouse position every time an Input.GetMouseButton(0) event occurs. I store the time of the click, and the current cursors position. I want to take that information, and playback the inputs on request.

I want to take the Vector2 coordinates and tell Unity to “Simulate a mouse down event at this Vector2 location at this time”.

Current attempted solutions:

  1. The only solutions I have found for this problem talk about using ExecuteEvents.Execute to fire off an event, but that requires you to specify a GameObject in which to trigger the event on, which I don’t want to store.

  2. The other solution was to implement methods from user32.dll and move the actual windows cursor and trigger mouse clicks from there. That currently only works if the user running the simulation again has the game window in the exact same location, and same screen as when the simulation was recorded.

  3. A 3rd solution was to have a script on every intractable object that monitors for a MouseDown event, but I want to avoid this solution at all costs.

This is what my class currently looks like:

    public class IntegrationTestBehaviour : MonoBehaviour
    {
        [SerializeField]
        public MouseInputInstance[] Inputs;
        private List<MouseInputInstance> InputRecord;
        private bool _recording;
        private float _recordingStartTime;
    
        public void RecordInputs()
        {
            _recording = true;
            _recordingStartTime = Time.time;
            InputRecord = new List<MouseInputInstance>();
        }
    
        public void PlaybackInputs()
        {
            for (int i = 0; i < Inputs.Length; i++)
            {
                StartCoroutine(HandleInputInstance(Inputs*));*

}
}

void Update()
{
if (!_recording)
return;
if (Input.GetKeyDown(KeyCode.Return))
{
_recording = false;
Inputs = InputRecord.ToArray();
}

if (Input.GetMouseButtonDown(0))
{
InputRecord.Add(new MouseInputInstance()
{
ClickTime = Time.time - _recordingStartTime,
CursorPosition = Input.mousePosition
});
}
}

private IEnumerator HandleInputInstance(MouseInputInstance mouseInput)
{
yield return new WaitForSeconds(mouseInput.ClickTime);
// TODO: Simulate mouse click at mouseInput.CursorPosition
}

[Serializable]
public class MouseInputInstance
{
[SerializeField]
public Vector2 CursorPosition;
public float ClickTime;
}
}
With the following CustomInspector
[CustomEditor(typeof(IntegrationTestBehaviour))]
public class IntegrationTestBehaviourCustomInspector : Editor
{
private IntegrationTestBehaviour _target;

public override void OnInspectorGUI()
{
base.OnInspectorGUI();
_target = (IntegrationTestBehaviour)target;
if (!EditorApplication.isPlaying)
return;

if (GUILayout.Button(“Start Recording”))
{
_target.RecordInputs();
}

if (GUILayout.Button(“Playback Recording”))
{
_target.PlaybackInputs();
}
}
}

2

I recently ran in to a similar situation.
As long as you are doing this on a PC, you can import user32.dll to fake the mouse position and mouse events.

Here’s a class that I found while looking for a solution to this.

using System;
using System.Runtime.InteropServices;

public class MouseOperations
{
    [Flags]
    public enum MouseEventFlags
    {
        LeftDown = 0x00000002,
        LeftUp = 0x00000004,
        MiddleDown = 0x00000020,
        MiddleUp = 0x00000040,
        Move = 0x00000001,
        Absolute = 0x00008000,
        RightDown = 0x00000008,
        RightUp = 0x00000010
    }

    [DllImport("user32.dll", EntryPoint = "SetCursorPos")]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static extern bool SetCursorPos(int x, int y);

    [DllImport("user32.dll")]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static extern bool GetCursorPos(out MousePoint lpMousePoint);

    [DllImport("user32.dll")]
    private static extern void mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo);

    public static void SetCursorPosition(int x, int y)
    {
        SetCursorPos(x, y);
    }

    public static void SetCursorPosition(MousePoint point)
    {
        SetCursorPos(point.X, point.Y);
    }

    public static MousePoint GetCursorPosition()
    {
        MousePoint currentMousePoint;
        var gotPoint = GetCursorPos(out currentMousePoint);
        if (!gotPoint) { currentMousePoint = new MousePoint(0, 0); }
        return currentMousePoint;
    }

    public static void MouseEvent(MouseEventFlags value)
    {
        MousePoint position = GetCursorPosition();

        mouse_event
            ((int)value,
             position.X,
             position.Y,
             0,
             0)
            ;
    }

    [StructLayout(LayoutKind.Sequential)]
    public struct MousePoint
    {
        public int X;
        public int Y;

        public MousePoint(int x, int y)
        {
            X = x;
            Y = y;
        }
    }
}

Edit:

To do a mouse click, call MouseEvent like this: MouseOperations.MouseEvent(MouseOperations.MouseEventFlags.RightUp | MouseOperations.MouseEventFlags.RightDown);

Of course, set the position first.