Best practices for parallel application + package development workflow

Unity Engine
,
1

I have a number of questions about other peoples’ experience developing multiple applications with dependencies on shared packages in private registries.

Essentially, we have a couple of Unity applications that use a lot of common code. I’d like to separate the common parts out into Unity packages, which we can readily reuse in other Unity applications. However, I’m struggling to optimize the workflow here because the Unity editor seems to be more focused on a use case of “this is your game, this is all of its code, and it is just one big thing in one place right here” rather than a focus on multiple applications with shared code.

Now, I can export parts of the project to Unity packages and reimport them into other applications, of course. If I want to create a Unity package as a standalone project, in its own source control repository, I have to create a completely new Unity project for it. This is fine but it seems like a lot of overhead, especially when the component I’m developing is simple. Still, I can make packages, I’ve got that down.

One of the two main issues I’m running into is this: Often when developing an application, changes might need to be made to its dependency components during development. In other words, developing an application is often done in parallel with development of some of the shared components it is using. For example, we have a component that acts as a network client to one of our desktop applications, and a couple of separate AR applications that use that component. But if I’m developing a new feature in one of the AR applications that also requires some new features in the network client, I have to be able to work on both the application and the network client package at the same time.

The problem is I can’t really see any way to do this smoothly. If I’m working on the AR application and need to make a change to the network client (I’m using this as an example), the only process seems to be:

  • Make changes to AR application.
  • Open a second Unity editor and open the network client package project.
  • Make changes to the network client
  • Re-export the network client package, being sure to select the correct files when exporting (there doesn’t seem to be a way to save the list of selected files when exporting a package, or to automate the export)
  • Switch back to the AR application project
  • Re-import the network client package
  • Continue working on the AR application.

But what I’d really like to do is something more like:

  • Make changes to AR application.
  • Open a second Unity editor and open the network client package project.
  • Make changes to the network client.
  • Magic happens and the changes are reflected in the AR application with no actions on my part
  • Continue working on the AR application.

Or, even more ideal:

  • Make changes to AR application.
  • Make changes to the network client directly from the same editor.
  • Continue working on the AR application.
  • Commit and publish everything, where magic happens to make everything end up in the appropriate repositories and registries.

Now, I’m aware of private scoped package registry support in Unity (which I have set up successfully, and this will be part of my second issue below), but using them actually seems to make the workflow even worse by adding an intermediate publishing step, changing it to:

  • Make changes to AR application.
  • Open a second Unity editor and open the network client package project.
  • Make changes to the network client.
  • Commit the changes to source control (required for next step)
  • Update the package version and re-publish to package registry.
  • Switch back to the AR project.
  • Update the package to the latest version with the new changes, using the Package Manage UI.
  • Continue working on the AR application.

This is really an awful workflow especially when the changes to the package are small, such as a minor bug fix or a few lines of new feature code, or when the changes aren’t complete and so a version patch number upgrade isn’t appropriate yet.

My question is: For organizations that have a number of applications and a collection of code shared among those applications, how do they deal with this? What are the best practices for package development, especially when application and package development happen in parallel? How can I remove the tedium?

My second problem involves scoped package registries. The issue I’m running into is, currently we primarily use GitLab as our source control service, and GitLab also provides package registry services. Unfortunately, GitLab’s package registry service isn’t fully compatible with Unity’s package manager: In particular, it works, but Unity is unable to display a list of the packages in a GitLab registry. The consequence is that even if you add the registry to Unity, you can’t use the nice Package Manage UI to browse and select packages. You still have to custom add packages from the registry and enter their names manually, which doesn’t provide any workflow benefit over simply not using the registry at all and just adding packages by git URL.

Now, there are other services, I think, that provide package registries compatible with Unity. “Verdaccio” is one that seems to come up a lot in forum discussions and tutorials. But I’d like to avoid adding yet another 3rd-party service to our already-gigantic collection of tools. I’m working with our dev-ops lead to try to find other solutions but we have no good ideas yet. The other reason it would be ideal to stick to GitLab is we already use the GitLab package registry features for other non-Unity things, and it is nice to keep everything in one place.

So, my question there is: Specifically regarding GitLab, are there any organizations that effectively use GitLab’s registry services with Unity and, if so, how do they do it? Are there perhaps Unity Editor extensions that can help manage GitLab packages and development workflows? Or other tools we can use that will let us stay with GitLab?

My end goal is to make development of multiple applications with multiple shared packages as smooth as possible, with the most optimized workflow we can get. Basically, as smooth as, say, VS is, where multiple projects may be open at any one time, worked on simultaneously, and built together in one step.

In my experience, tedious workflows ultimately lead to poor quality products over time, as developers (including myself) who want to focus on coding start to invent fragile shortcuts around troublesome procedures. So, I’m trying to nail this down as much as possible now before our projects get much larger.

I hope this makes sense, sorry for the length.

Thanks!!

2 Likes
2

First problem, simple solution:

  • put the package in its own repository
  • yes, just the package folder, not the entire unity project
  • clone the package repo somewhere on your disk
  • yes, in its own folder, not anywhere inside a unity project
  • open any unity project
  • package manager —> add package from disk —> select package repository folder

Now you can make changes to your package from any unity project the package is embedded in, and all changes are done in the same local repository.

Cannot comment on the second problem.

1 Like
3

PS: if you need assets in the package be sure to check package.json flags, one of them controls whether package assets are shown in picker windows such as material or texture picker.

1 Like
4

Thank you so much for this. I tried that and it worked; although the process was slightly different (could be because I’m using 2020 for this project, for Lumin support) and I was prompted for the package.json file rather than the directory. The end result was fantastic, though, and solved the editing problem. Also surprisingly, it didn’t matter where the package.json file was in the directory structure, it just worked.

This makes the development workflow perfect, but adds kinks to the package adding workflow, which becomes (when a dev wants to add a package):

  1. Externally, clone the git repo of the package.
  2. In editor, add package from disk.
  3. Browse to the location of the git repo.
  4. Select package.json.

Which isn’t too hard, really, except I’d still like to remove any external steps and, if possible, have a browseable list. To the first point, I tried instead adding the package directly from the git URL, although I’m having issues determining the correct location of package.json, which I’ve asked about in a new post .

It’s probably possible to write an Editor plugin to browse, clone, and install packages from our private git repos; I know interaction with the package manifest and package manager UI is possible because there’s a couple of 3rd-party package server managers on the asset store that do that. It’s something I’d like to avoid adding to my todo list, though.

In a situation where there are many package and application repos on a git server, where applications are developed using those packages, do you know of any way to smooth out adding packages to applications to eliminate the clone and browse-to-file steps, so that all of the actions can be done within the Unity editor?

Well, I think the second problem becomes moot in cases where doing development on the package itself is desired, since any packages installed via a registry wouldn’t be bound to a git repository. So it looks like there isn’t really a place for the package registry here, it’s only appropriate for packages that don’t need to be actively worked on. Kind of obvious in retrospect.

Thanks again, really appreciate it. You were super helpful.

1 Like
5

I was going to say “import package from github” but yeah, I remember I had issues with that too. I think one being dependencies, you cannot have package A depend on “githubbed” package B. Or at least it won’t automatically add it to the project.

To fullfil your requirement I believe the way to go is to set up a custom registry for your own packages, so that they work just like Unity’s packages in the Package Manager.

6

Ah, thanks for the dependency heads up, didn’t think of that. Yeah that would definitely be an issue. I’ll have to experiment though. NPM allows git URLs as package identifiers in the dependency section of the manifest, so my theory is if I use the git URLs instead of the package names in every package’s dependency list, maybe UPM relies enough on NPM’s behavior that it would just grab everything recursively from git. The only problem is you lose smooth semver support, versions are identified by git tags instead of verson numbers and the nice semver update logic that NPM does no longer applies to packages linked via git URLs (essentially the version is in the URL, so it becomes an inseparable part of the package “name”). If I try that, I’ll post back with the results.

I’ll have to see if I can figure out a way to automatically clone git repos for packages installed from custom registries. That’s the dream, having both the “add from disk” developability (…?) + the package registry convenience. Few steps closer now.

Thank you again. If I find any solution to that on my own I’ll post it here.

1 Like
7

This thread helped me with the “convenient editing of packages” problem. Thanks!

However, @JasonC_SN or @CodeSmile have you also solved some of these limitations?

  1. The IDE (Rider) is not happy about the namespaces, probably because the scripts are now under Packages/, and it wants to “move” all scripts to global namespace (and scripts in Editor folders into “Editor” namespace). The warning goes away if I specify “root namespace” in the asmdef to be the same as namespace itself, but this feels wrong.

  2. I need some of the assets of the package to be located at data path - for example a settings scriptable object asset. The code looks for directory somewhere under data path (Assets) and then tries to load the setting asset. If it is not found (or the directory), the directory as well as the asset are recreated. This is aimed for convenience of the target user if they delete the asset by mistake, or other reasons. I understand that with this solution, the repo lives outside data path and so I would need to change the path to direct location of the asset (if that would even work?). Then before deployment of the package to asset store, I would need to change the logic back to “normal” - data path driven approach. Also feels wrong.

Thanks again for great solution, and if you have any ideas, please share.

8

Another limitation I discovered is when I started building UI Toolkit visual trees. Apparently, the UI Builder can’t access files under Packages/, so I go around this by moving the Resources folder (containing the uxmls) to Assets/ folder, and back under Packages/ if I need to commit.

Any ideas for better solutions? Thanks!

9

I am so annoyed by the fact that there’s no sane way of establishing the package editing environment other than the separate Unity project. In theory, if your package has no dependencies besides the Unity, you can create a blank solution, resolve UnityEngine and UnityEditor dependencies and move on, but, say, if you’re forking some of the Unity packages to improve them, its borderline impossible since the dependencies would kill your effort.

Right now I’m working with Splines package to get the geometry generation for my needs, it seems like to clone the fork as embedded package in the isolated project is the only option.

1 Like
10

Could you clarify?

I mean you have to have a Unity project to be able to test a package. Do you want to write and test a package without the package being consumed by a Unity project?

Nobody does that. I would say. Probably not correct, sure. On the other hand, those who do modify Unity packages may have forgotten that they don’t have to but can rather be a consumer of the package. Or copy an essential class, and modify it heavily, like I did with InputManager and PlayerInputManager in order to get a splitscreen (couch) player input class that is manageable, eg 200 lines instead of 2,000.

It’s really not recommendable to fork and modify Unity’s own packages besides the occassional hotfix or patching some class/method to be public. Simply because you never know what direction Unity will steer this particular package towards and at what pace, and then sooner or later you end up with this sort of situation:

You may have to keep using “Unity package v1 (your fork)” while the package is now v2 and has been heavily refactored, so not possible to merge your fork changes. But only v2 is compatible with the editor version that you have to upgrade to because otherwise you can’t build for a given target platform, or use a Cloud Service the game requires, or … you get the point.

Forking and modifying Unity packages is a sure way of moving yourself into a technological dead-end!

11

Ah! Looks like someone was having similar issue.

I am trying to find how to create a custom unity package in Rider, but no solution yet.

How are unity packages created at Unity? Manual process?

What IDE do you use to edit those project? I’m in Rider, but I have VScode and VSSTudio.

Can a .csproj and .sln be created to manage that costum unity package?

image
image902×693 49.7 KB

Is there a solution template I can download to create unity package? I couldn’t find one here: Available templates for dotnet new · dotnet/templating Wiki · GitHub

image
image275×233 3.23 KB

@CodeSmile

12

Yup. The docs even has steps like " Open your preferred text editor and create a JSON file called package.json in the root of the package folder.". So pretty manual.

You can use whatever IDE, doesn’t really come into it.

Unity packages are C# libraries, but their own thing. Unity (for now) generates .csproj files and .sln files for your IDE to parse, but doesn’t use them at any point during compilation.

1 Like
13

Exactly, I am also trying to setup a dev environment for my own package ATM.

14

Ah ok, that’s a bummer.

Thank you @Baste

15

Rider does understand the package.json format if I recall correctly, when editing in a Unity project solution. So at least it should give you autocompletion suggestions just like it does for .asmdef files.

Personally, I have a template folder with a package.json, the default folder structure (Editor, Runtime, Docs, Tests) and some default files (readme, license, changelog) so I can quickstart a new package. You’ll also find similar templates if not package creation tools on github.

16

ok, better than nothing.
I just can’t have a .csproj for the package I guess.

Thank you!