CI / CD pipeline for Unity (github actions? cloud device farm?)

We are developping a mobile app in Unity and are asking ourselves, what would be the best way to set up a continuous integration / continuous delployment (CI / CD) pipeline.
I’ve seen tutorials how to setup github workflows to automatically test each push to github on a couple of devices (e.g. Windows, Mac, Linux). However in my case the performance on a large number of mobile devices would be the most interesting test. I don’t see how I can do this using github actions.

Has anyone a recommendation how to setup such a pipeline?

I know of cloud device testing services such as Kobiton and AWS. But I’d like automate the pipeline such that everything runs automatically after a push to some branch on github.

Here is my “work-in-progress” on the topic:

There seem to be some non-Unity-related tools that aggressively pop up, when you google the above mentioned problem. E.g.

However I didn’t test them, so I cannot say much about them.

Turns out Unity has a package called “Automated QA” which

  • to record button / screen click events to generate an automated play through test
  • Once a recording has been generated, you can generate a “full test” from the recording (also using the automated QA package
  • now open the Unity internal Window → General → Test Runner
  • here you can find your earlier recording as a kind of unit test
  • you can run the test in the editor and on device (e.g. Android)
  • in the process Unity creates a “TestRunner.apk” which one can install on any number of cloud (Android) devices

TODO:

  • run play through test from command line
  • get results as text file from device (currently the TestRunner only creates a folder with screenshots but no logs)

Some more useful links I found