Trying to Automate Windows Calculator from Unity crashes unity

Unity Engine
,
1

I’m trying to automate the windows calculator from unity

I put these dll’s into unity’s plugin folder not sure if that’s what you’re supposed to do.
from c: program files/referenced assemblies/Microsoft/ framework /3.0
uiautomationclient.dll
windowsbase.dll
uiautomationtypes.dll
uiautomationprovider.dll
uiautomationclientsideproviders.dll
uiautomationprovider.dll

The code I’m running is

using UnityEngine;
using System.Windows.Automation;
using System.Windows;
using System.Diagnostics;

public class main : MonoBehaviour
{

    private void Start()
    {
       Process calculatorProcess = Process.Start("Calc.exe");
       ElementFromCursor();
    }


   private void ElementFromCursor()
    {
        // Convert mouse position from System.Drawing.Point to System.Windows.Point.
        //System.Windows.Point point = new System.Windows.Point(Cursor.Position.X, Cursor.Position.Y);
       Point point = new Point(1500,1500);
        AutomationElement element = AutomationElement.FromPoint(point);
        //UnityEngine.Debug.Log(element);
        //return element;
    }
}

The Error is:

  • Assertion: should not be reached at …\mono\metadata\marshal.c:7146
2

Mono (which Unity uses to run C# code) doesn’t really support COM interop well. You’re running into one of many of its limitations. If you want, you can report a bug on it but it’s unlikely to be fixed in a a timely manner as this is a very niche use case.

You might have more luck if you build your project using IL2CPP scripting backend.

3

Thanks for the tip. I wasn’t expecting any help since it’s such a niche case so I was glad to see a response. I tried using IL2CPP but same result. Editor and Build same result on mono and on IL2CPP. I don’t get any errors in the editor or player log so there’s nothing to research.

4

When you say same result on IL2CPP, what do you mean? That assertion is coming from inside Mono, so it cannot be hit by IL2CPP.

5

Hi. I changed something in the player settings from mono to IL2CPP is that what you meant to do? By “the same result” I meant it crashed. I did not see the mono error or any error in the logs on the IL2CPP build.

6

It crashed the build? Interesting, I wonder what kind of crash that is. Any chance you could attach a debugger to crashing IL2CPP process?

7

It made the build without an error. Upon running it opens the calculator, then closes the application. The editor does the same thing. It does not hang anything, but actually closes the entire editor or the standalone. I’m assuming this is caused by the AutomationElement.FromPoint call. If I take it out it does not close/crash.

8

How do I attach the debugger? there are some player settings and some settings in the build panel?

9

I did a developmental build, script debugging, wait for managed debugger? not sure what that is.
In the player settings I put on full stack trace on everything.
But no new lines in the log. The last log lines of the player crash are…
Begin MonoManager ReloadAssembly

  • Completed reload, in 0.895 seconds
    Initializing input.
    XInput1_3.dll not found. Trying XInput9_1_0.dll instead…
    Input initialized.
    Initialized touch support.
    UnloadTime: 2.270817 ms
10

Okay now I got something in the editor that said it couldn’t do the build. For some reason the editor switched back from IL2CPP to mono after it crashed. So now I’m unsure as to whether I had really made an IL2CPP build before.

IL2CPP error for method ‘System.Int32 MS.Internal.Automation.UiaCoreApi::RawUiaGetRuntimeId(MS.Internal.Automation.SafeNodeHandle,System.Int32[ ]&)’ in assembly ‘C:\Users\dan\desktopscanner\Temp\StagingArea\Data\Managed\UIAutomationClient.dll’
Additional information: SafeArray element type None is not supported.

11

Indeed, this looks like just COM interop hell - those DLLs call system COM APIs and neither Mono nor IL2CPP are able to handle that. I think your only option is to call those COM APIs from a native plugin if you’re determined to make this work.

12

So far after many years I haven’t run into something that unity can’t do until now, but it seems like for a good reason. UIAutomation seems like an unreliable mess the more I read about it.

Thanks for helping me determine the root source of the problem which is helping me avoid this library.