Setup for scoped registries (private registries)

rz_0lento · October 25, 2018, 11:24am

2018.3.0b7 added following:

We already got informative post about how to use git from package manager here:

Any chance to we could also get some brief introduction on how to setup scoped registries (private registries)?

okcompute_unity · October 25, 2018, 11:47am

Sure. Here it is!

Introduction
The concept of scopes is borrowed from npm (https://docs.npmjs.com/misc/scope), it allows a subset of packages to be hosted on a registry other than the default.

Configuration (manifest.json)
Project manifests (manifest.json) can now contain any number of scoped registry configurations in the scopedRegistries attribute.

The scopes attribute of each configuration is an array of strings that can be mapped to a package name in two ways:

As a namespace. This type of configuration assumes that packages follow the Reverse Domain Name Notation (Reverse domain name notation - Wikipedia). In other words, a scope such as com.unity is the equivalent of com.unity.*
As an exact package name match.

When deciding in which registry a given package should be searched, Package Manager will settle on the scope that is the closest match to the requested package name.

The snippet below demonstrates how scope selection works:

{
  "scopedRegistries": [
    {
      "name": "Main",
      "url": "https://my.company.com/registry",
      "scopes": [
        "com.my-company", "com.my-company.tools.packageX"
      ]
    },
    {
      "name": "Tools",
      "url": "https://my.company.com/tools-registry",
      "scopes": [
        "com.my-company.tools"
      ]
    }
  ],
  "dependencies": {
    "com.unity.cinemachine": "1.0.0",
    "com.unity.2d.common": "1.0.0",
    "com.unity.2d.animation": "1.0.0",
    "com.my-company.package1": "1.0.0"
  }
}

With the above configuration:

com.my-company.packageA. will be fetched from Main because it is in the com.my-company.* namespace.
com.other-namespace.packageX will be fetched from Main because its name is an exact match. It also matches com.my-company.tools.* (Tools) but that is less specific.
com.my-company.tools.animation will be fetched from Tools because it is in the com.my-company.tools.* namespace. It also matches com.my-company.* (Main) but that is less specific.
com.other-company.packageA will be fetched from the default registry because it matches none of the configured scopes.

Why “scoped” registries?
It’s to ensure determinism when deciding from which registry a package should be fetched. By using scopes, a package will always be mapped to one and only one registry, guaranteeing a consistent result regardless of network conditions.

Supported Registries type
Unity Package Manager supports npm protocol based registries. You can use any off the shelves npm registries server and it should work.

These are the registries that we know should work

Verdaccio Probably the easiest server to configure

Search all package limitation

Some npm registry server does not support the /all route to search all packages (e.g. Artifactory). Displaying scoped registry packages as part of the “All packages” list relies on scoped registries to support the old npm API protocol, which has an HTTP endpoint that returns the metadata of all published packages (eg: https://registry.my-company.com/-/all). When a registry does not support the old protocol, packages from that registry will simply not be displayed in the UI. However, this limitation does not apply to package resolution, so packages from scoped registries can still be manually added to the project manifest.

Limitations

In the Package Manager window, the “All packages” tab will now display and allow you to install packages found in scoped registries. But, they will be mixed with the default Unity registry. There is no differentiation for now. In future releases of the UI, the source and the scope of each package will be made clearly visible.

Thanks for trying out this feature. Let us know if you have any issues. Look for a production quality release in 2019.1.

Regards,

Pascal

M_R · October 25, 2018, 5:28pm

Hi,
how does it work with registries/packages that requires auth? it picks up whatever npm login you do in the terminal? or requires env variables / cli startup?

also do you plan to integrate with the unity account management (so it picks the credentials I am logged in unity with)?

okcompute_unity · October 25, 2018, 7:03pm

Hi @M_R ,

The Package Manager does not support authentication for now. Your repository needs to be public. One thing you can do is limit access to people inside your LAN.

If you have a use case that is common, we could look into adding this feature. Can you expand a little bit more on why you would need auth on a local/private registry?

Regards,

Pascal

M_R · October 26, 2018, 9:32am

to prevent unwanted access to the packages, the same way we have auth required on bitbucket (ssh, it works btw).
also we need access at least from:

our office
our homes (or anywhere we may be, i.e. if we need a urgent hotfix in the weekend)
cloud build

the best pattern would be to link it with our organization for Unity account, so anyone (plus cloud build) can have access without modifying either env or manifest.json.

for the main registry, I can access private stuff by modifying the manifest, but it feels hacky (and doesn’t support multiple users without gitignoring it):

{
    "registry":"https://blablablabla",
    "always-auth":true,
    "email":"hardcoded@email.com"
    "_auth":"<hardcoded basic auth>"
    "dependencies":{...}
}

okcompute_unity · October 26, 2018, 12:01pm

Thank you @M_R for the feedback! Will transfer your request to our product manager to validate priority.

Regards,

Pascal

Adrian · November 1, 2018, 3:54pm

Easiest would be if Unity supported the standard npm authentication.

i.e. a user would log in with npm adduser REGISTRY, which updates the .npmrc file in the user’s home directory and Unity would pick up those credentials.

This would mean existing private npm registry setups could be used with Unity without having a separate authentication process.

okcompute_unity · November 1, 2018, 4:50pm

Hi @Adrian ,

I took note of your suggestion.

Thanks,

Pascal

M_R · November 2, 2018, 3:37pm

as @Adrian says, plus the ability for npm adduser ... in the cloud build

okcompute_unity · December 4, 2018, 4:43pm

@M_R I just wanted to give you a follow-up on this feature request. We analyzed it, and it will require a bit of work to provide a good user experience (persist credentials, env vars, login window, etc.). The complexity is a bit greater than I would have anticipated. I hope we can implement this feature sometime in 2019. We have a couple of higher priority issues to address in our backlog before attacking this one. As for your request to support Unity auth on scoped registries, it is not something we have in our roadmap. Sorry.

@Adrian Even though Package Manager is compatible with npm web API, it is not the same technology underneath (e.g. we don’t share the same configuration scheme for scoped registries)

Regards,

Pascal

dzamani · December 7, 2018, 5:43pm

Hi @okcompute_unity ,

I was wondering if you could ask about another way to do auth with packages. The idea would be to have a callback everytime the package manager try to access scoped registries. That callback would be called for each scoped registry and should return a boolean saying if it has access or not to it.

Something like this:

bool HasAccessToRegistry(string registryUrl)
{
    var authToken = FetchToken();
    if (registryUrl == "someurl" && authToken.IsValid())
        return true;
    return false;
}

The FetchToken would be something the user will develop.

Now I know the management of the registry isn’t done at the package level but somewhere in the engine (if I’m not wrong) so having that callback may not be that easy based on your architecture but that would be perfect for us to do what we want about these registries.

Auth is the main reason why we can’t have public registries since we can’t share our code but having LAN only packages is preventing us from remote working (well not absolute prevention but still it’s very painful to grab each package manually from their git repo when you are at home).

Anyway if you could ask about the feasibility of this solution, that would be great, thanks!

okcompute_unity · December 7, 2018, 7:42pm

Hi @dzamani ,

Thank you for your suggestion. We have a well-defined plan to implement this feature. This is only a matter of prioritization. If you have an enterprise support account, you can contact your assigned account manager to express the value for your company to make this feature available sooner.

Regards,

Pascal

nhold · December 19, 2018, 6:20am

Will there\could there be support for just a git repository representing a registry? I do this method with my custom package manager and it makes it really easy for anyone to make one.

okcompute_unity · December 19, 2018, 3:10pm

@nhold Can you expand a bit more on your setup? Do you have examples? Or a public repository that you can share?

Thank you!

Pascal

nhold · December 21, 2018, 10:21am

Obviously my setup will be a lot simpler but you can see the repo for the package manager itself here:

https://github.com/nhold/ubPackageManager

Essentially the idea is a ‘registry’ is just a git repo with lots of Package Definition Files which is defined as:

{
    "name": "ubGridArray",
    "versions": [{
        "version": "1",
        "branch": "version-1"
    }],
    "description": "1D array as 2D array.",
    "location": "https://someurl/ubgridarray.git",
    "parentDir": "Bifrost",
    "childDir": "ubgridarray/Assets/Plugins/ubGridArray"
     "dependencies": [
        "ubConfig"
    ]
}

So to read from the registry you just pull\clone it and read the json files to populate the data.

okcompute_unity · December 21, 2018, 1:33pm

Interesting. We have no plan to support such a feature. I just don’t want to give you false hope. But will see. If hosting out of the shelve npm registries turns out to be too hard/complex for people, we could consider this alternative approach.

Btw, were you aware that the Unity Package Manager supports Git URLs project dependencies? Git support on Package Manager

Thanks a lot for sharing. This gives me another perspective

Regards,

Pascal

dtaTrifork · January 15, 2019, 5:41pm

What are the rules about version matching within scoped registries?

Traditionally we would base rules on the NPM protocol found here: https://docs.npmjs.com/files/package.json

Suppose you have the following packages:

A in versions 1.0.3, and 1.2.0
B depends on A version 1.0.3
C depends on A version 1.2.0

Is this something we can expect the package manager to be able to handle and correctly isolate? As far as my testing illustrates the package manager will default to the highest versioned dependency.

Finally, the dependency on package A seems to be a hidden package within the Library/PackageCache along with the remaining packages. But said dependency will not be illustrated within unity.

okcompute_unity · January 15, 2019, 6:37pm

Hi @dtaTrifork ,

The package manager conflict resolution algorithm will always choose the highest version number requested unless the package is a root dependency (i.e. you explicitly added the package to your project). We are working on documenting the conflict resolution algorithm. Coming soon.

The Package Manager UI does not show transitive dependencies. This is something we have on our roadmap.

Regards,

Pascal

giacomohomunculus · February 7, 2019, 8:21pm

Hi,
I am setting up Unity Package Manager to work with a private npm registry.
I am getting an error:

An error occurred while resolving packages:
Project has invalid dependencies:
mypackage: self signed certificate in certificate chain

I can pull the package from the registry using NPM fine, how can I specify the cafile for the Unity package manager?

okcompute_unity · February 7, 2019, 9:38pm

Hi @giacomohomunculus ,

I’m sorry. The answer is not yet. This feature was recently requested HTTPS Communications Options - Unity Engine - Unity Discussions . We set the priority for this feature very high.

Regards,

Pascal