Feedback Wanted: Unity Downloader - Powering the Editor as a continuous integration dependency

We would love to hear your feedback about the blog post we wrote about one of our internal tools that make it possible to grab Unity versions through CI:

Unity Downloader: Powering the Editor as a continuous integration dependency

What did you think about the blog post? Does this type of tool relate to problems you currently have as a Unity user? How could we make your CI work more fluent and enjoyable?

11 Likes

That was a nice blog post and it gave us some hint on your work and how your workflow works internally with CI. That's always nice to ear that kind of things.

For me, this tools could be useful to retrieve specific version when we build our apps with CI. My main and sadly unsoluble issue is to retrieve a specific version of Unity for a specific project. If it's not installed on the building machine it's currently not possible (without some effort) to retrieve it. I was waiting for Unity Hub to release some specific CLI version and in this way allowing me to retrieve specific version but your Unity Downloader tools seems to do (more or less that kind of stuff)

Cause basically what I'm doing currently to retrieve and start the good version is to let my Gitlab-runner start the job, check the project on the branch where the job start. In this project I check the "ProjectVersion.txt"'s content generated by Unity for this one and start the good Unity Editor accordingly.

Maybe it could be used separatly or maybe you could "plug" it behind your Unity Hub stuff to allow any CI devs to retrieve specific Unity's version.

1 Like

This is exactly what we're doing as well. Any time we update the project to a newer Unity version we have to remember to also install the new version on the GitLab Runner, otherwise the build will fail. Would be nice to finally automate this completely.

I like the idea of being able to just specify a project and let the downloader find and download the correct version. For this feature it would be good to have a couple of modes:

  • Just download the determined Unity version
  • Download + open project
  • Download + open project with specified launch arguments
  • Download + return path to Unity.exe or install folder

I also think that it would be nice to integrate this into Unity Hub in some way. When the project is updated or I check out a different branch that uses a different Unity version and I try to open the project without having that version of Unity installed locally, Unity Hub should offer to download that version of Unity, instead of running the script conversion tool.

3 Likes

Unity hub restricts editor versions to a small subset, so when I wanted to get the same project behavior of some old version I had no choice other than download the closest editor version and hope there will be not so much differences and I can handle them in one pass. Otherwise, I could easily upgrade my project with a small steps.

1 Like

Thank you for your feedback.
This is for sure one use case that I suspected people might need tooling for.
The current unity downloader does have support for specifying an existing Unity project, and as you mentioned, takes a look in the ProjectVersion.txt and downloads that exact editor it was imported with.
So I would like us to be able to deliver something that would solve this for our end users as well.

Edit: Posted a previous reply with my other user account so deleted and recreated it. Sorry about that :smile:

Our workflow is basically as follows:

  • our build system config contains mapping of our builds to unity version (SuperGame1 ios version -> unity 5.5f1, SuperGame 1 windows version -> unity 5.5p5, MegaGame2 any version -> unity 2019.1f1)
  • before build it checks if unity version is installed and if not downloads it
  • we've been using u3d with great success. and before our custom scripts.
  • This works for both mac and windows machines

This system has some downsides:

  • There is no way to add packages. If we suddenly decided to add xbox platform we need to delete unity from boxes and then u3d will download it from the start. This is quite seldom but still an annoyance.
  • There is no cache right now - every box has to download it by itself. This is seldom too.
  • There is no way to add Android build tools using u3d right now. This is very annoying, because this is actually is a great feature, and when using unity >=2019 we don't need to manage multiple android sdk/ndk on machines
1 Like

The Hub only shows the latest version of each major release i.e. latest 2018.3.x, latest 2019.1.x etc. But on the same window, they provide a link to the archive where you can download all version.

1 Like

This seems like it would be best served as an add on to the Unity Hub, and Unity Cloud Builds. Focus should be on improving your existing CI tools like Unity Cloud Builds, which to an extent has this feature sort of implemented as is. One thing that's a pain (that others mentioned) is downloading a version of Unity that is buried within the Download Archive, and not directly available through the Hub, a simple CLI would be great, or better yet a proper search filter within the Hub. I enjoyed the blog post, and hope to see this integrated into existing tools instead of being yet another dependency I have to install.

1 Like

Thanks for all the feedback so far. It is really useful.
This solution came from an urgent need to get something in place, since packages were already upon us a year ago and we needed a single solution internally for delivering editors our different teams had for their CI.
It was very much a "Let's see if this is useful for us" on my part which is why I spent my 20% time on it.
Almost a year later it turns out it was very successful and so I wanted to reach out and see with you, our users, if similar functionality would be of use to you as well.
Based on reactions so far that seems to be the case.
I have gone into this with very few assumptions for what a public solution for this would look like.
If it makes the most sense to add this type of functionality to the Hub, then I will spend my time towards that.

So a question:
Would local editor caching be useful?
What I mean by that is for example a server sitting somewhere in a closet where all editor versions previously downloaded sit and will be served when someone at the location selects to install it.
Which is basically what we do with the current downloader. We have a cache sitting on premise where our build VMs are located so we do not have pre-installed editors on the machines. We just download and sets them up at the start of every CI job.

Concerning my part, the cache is definietly useful because we have only one building machine and testting each profile to build our app take something like 10-20minutes (12 profiles to recompile each time someone is committing something).
We want to avoid adding more time to this pipeline already quite long (can't buy others machine to do this for now). Just because we use few Unity version, we keep them on the machine forever (almost) and we only reset the project content because it's logical to have something clean when the runner start.

Anyway the runner clear automatically each element which is not pushed on a branch (like Library folder obvisouly) that's why I can say the project is clean when a job start.

Thanks again for your time and your return based on ours.

2 Likes

I rarely post here but the blog post actually intrigued me enough to do it (albeit for the first time)

The CLI is tremendously needed if not just for the editor download, since it would be able to run the downloading and the compilation of the Unity Editor in a cloud environment under our control, but also because it could aid the total CI/CD process in total.

As it stands now, the CLI aspect of downloading the editor would be fine to bundle with Hub for the user experience part of it (kind of shipping both the GUI and an "CLI" version of the gui in one is a good solution for most users)
That is how some applications do it, I remember Butler (the itch.io CLI for shipping builds) - used to be bundled with their Game Store application.

However I see a greater potential of this tool not only to download the Editor, but also resolve the Packages from within a project we try to open. (Even maybe preheat the GUI editor ?)

If Unity are serious about Packages being the way forward, we should also be able to "move" the Unity Asset Store *.unitypackage (packages) - over to this system.
Therefore resolving any Unity Asset Store dependency enabling us to do all of our builds within a CI/CD environment.

It would be a HUGE enabler for me because I would be able to ship my builds just from within my repository without ever having to boot up my PC to initiate a build - simply just push to my master/trunk branch and the CI/CD pipeline would be able to push it for me.

1 Like

Asset Store Tools developers would benefit immensely from the ability to test installs and Editor Unit Tests as new versions are released.

In fact, all Asset Store assets could benefit from this to some regard. Some visual reporting mechanism could be used for art assets, provided a system could be devised to include a manifest describing scenes to test (and take framebuffer captures of for reporting/comparison).

I would highly recommend posting to the Asset Store Publishers forum as well as we represent a segment of your user-base that actually shares quite a bit in common with your own internal development processes and requirements.

2 Likes

We faced the same issue. So we created a tool in Ruby we called uvm which in the beginning allowed to simple set symlinks to the “active editor” on MacOS

I rewrote this tool in rust last year and reverse engineered Unity Hubs install logic.
https://github.com/Larusso/unity-version-manager

I created a small web service to fetch the latest release of the editor and store it in a github repo.
https://github.com/Larusso/unity-versions-service

Last but not least I wrote a JNI wrapper to call the tool/lib from Java/gradle

https://github.com/wooga/unity-version-manager-jni/pulls

https://github.com/wooga/atlas-unity-version-manager

Most of it is still under development but these tools run our CI system for over a year now.

1 Like


That should be done now, thanks for the tip :)

And in general:
Thanks for all the great feedback everyone, keep it coming, I can take it.
It is of immense value to us to know your thoughts and concerns on this subject.
So many of you sharing your own solutions is also very appreciated.

1 Like

First, thanks for sharing that tool (whenever it will be available)!

I was thinking but would it be possible to have some authentication or something in order to also download consoles components ?
Since I guess you don't need auth at Unity we don't have any way to automatically download those component at the moment. It would be the perfect time to think about a way to make it possible.

About the tool itself, not much haven't already been asked but I will ask for the simplest thing which is a way to download any Unity with a command line while also being able to check which versions and which components are already installed. Also (maybe it's already the case) but it should work on Windows and MacOS.

Thanks!

1 Like

Sounds like a great tool.
Over the years I got to work on different projects, where we needed to test our content in different Unity versions.

Games:
To check if we can upgrade the projects to new editor versions.
Then we decided if the hassle of upgrading them is worth it.

Plugins:
To check if plugins that we developed can be used in new editor versions, and if we need to do adjustments for them to work.

So a way for us to check if there's a new editor, install it, and test a project could be great

1 Like

Thanks for the post, it's great to hear about the problems you guys face internally and how you guys are addressing it. It is also extremely relatable.

Our company does CI in the cloud with stateless VMs so every time we run a build we have to install unity.

Currently, we are pulling down the installation files and just installing them via the command line. This whole process is automated (and changing the version is simply updating a variable in our CI pipeline) except for one step - the need for having to prepare those installation files in the first place. This means someone actually has to go and download the installation files and store them in a place for ease of access.

So far this is not too bad, downloading and installing Unity and the editors takes us about ~50seconds in total and the manual step of having to upload the installer files are only ever required if a project requires a new version of unity and we are looking to automate this soon.

However that being said, it'd still be great if Unity can offer a first-party tool that will do this. My concern would probably be the speed of the download, as long as it is easy to cache that it'd probably be something we would like to adopt.

One last issue we have which may be off-topic, but is Unity looking at offering build licenses for companies? That is one of the biggest issue to us scaling our CI system.

1 Like

This would be wonderful to have. Looking forward!

1 Like

This tool would be extremely useful for our CI in Azure DevOps. We are currently downloading the packages seperately, but this would ease a lot of headache for us. Especially since it seems that this tool can automatically detect the Unity version to download based on the project. I'm super excited to use it! :)

1 Like

Thanks for the detailed explanation.
Regarding licensing, we are currently working on a solution that would help with this. I don't have more details than that for the time being :)