Git support on Package Manager

spaceemotion · January 9, 2019, 7:12am

@pedro_unity Hey, thanks for the reply.

I am running an ssh-agent in the background and the PID env-variables are being set properly. For some reason Unity does not seem to pick up on this though.

This is in my .bashrc file, I’m starting it by opening the git bash on windows (Havent tested this on my mac yet):

env=~/.ssh/agent.env

agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; }

agent_start () {
    (umask 077; ssh-agent >| "$env")
    . "$env" >| /dev/null ; }

agent_load_env

# agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2= agent not running
agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?)

if [ ! "$SSH_AUTH_SOCK" ] || [ $agent_run_state = 2 ]; then
    agent_start
    ssh-add
elif [ "$SSH_AUTH_SOCK" ] && [ $agent_run_state = 1 ]; then
    ssh-add
fi

unset env

The contents of that .env file after authenticating is:

SSH_AUTH_SOCK=/tmp/ssh-x7j1vRXCkHpc/agent.17496; export SSH_AUTH_SOCK;
SSH_AGENT_PID=10644; export SSH_AGENT_PID;
echo Agent pid 10644;

pedro_unity · January 9, 2019, 1:32pm

Hi @spaceemotion ,

That’s weird, let me dig a bit more and get back to you.

Cheers,
Pedro
Unity STE

maximeb_unity · January 9, 2019, 4:17pm

spaceemotion:

@pedro_unity Hey, thanks for the reply.

I am running an ssh-agent in the background and the PID env-variables are being set properly. For some reason Unity does not seem to pick up on this though.

This is in my .bashrc file, I’m starting it by opening the git bash on windows (Havent tested this on my mac yet):
env=~/.ssh/agent.env

agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; }

agent_start () {
    (umask 077; ssh-agent >| "$env")
    . "$env" >| /dev/null ; }

agent_load_env

# agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2= agent not running
agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?)

if [ ! "$SSH_AUTH_SOCK" ] || [ $agent_run_state = 2 ]; then
    agent_start
    ssh-add
elif [ "$SSH_AUTH_SOCK" ] && [ $agent_run_state = 1 ]; then
    ssh-add
fi

unset env
The contents of that .env file after authenticating is:
SSH_AUTH_SOCK=/tmp/ssh-x7j1vRXCkHpc/agent.17496; export SSH_AUTH_SOCK;
SSH_AGENT_PID=10644; export SSH_AGENT_PID;
echo Agent pid 10644;

Hi @spaceemotion ,

While your script may successfully start the ssh-agent in the background, the exported environment variables are only available to the bash shell running the script and to any child process it launches from that point on; environment variables from parent processes, unrelated processes, or previously created child processes cannot be changed by your script. Further, even if the script is located in .bashrc, it only gets executed in bash shells. Launching programs (both in Windows and macOS) does not use a shell, so the script is not executed. On the other hand, if you open a bash shell first, then “manually” launch the Unity Editor from that shell, it should work.

In other words, if you use the script by running it once, then you launch Unity by double-clicking its icon or using the Hub, the Unity Editor process won’t have these variables set so ultimately the Git process invoked by Unity won’t have them either.

That said, I see a few possible workarounds.

You can create a bash script to launch the Unity executable or the Hub (caveat: the Hub must not already be running, otherwise that will just activate the existing process and it won’t have the newly exported environment variables)
Or since you’re on Windows, you might try to invoke SETX to permanently set the environment variables in the Windows registry. However you need to ensure the Hub process is not already running, or that it gets restarted if it’s already running. This solution would work best if you can ensure that the script runs once on startup and that the Hub is not started with Windows.

The last suggestion would have the benefit of seamlessly integrating the ssh-agent so it works as you expect it to.

spaceemotion · January 9, 2019, 7:19pm

@maximeb_unity hey thanks for the suggestion. The second one might just what i needed to also fix other ssh-agent related issues I’ve had with git clients (Tower, specifically). Will try this out asap! Cheers!

spaceemotion · January 9, 2019, 11:19pm

@maximeb_unity

Good news! I got the whole thing to work by appending the following to my .bashrc:

# Expose env variables to native windows
setx SSH_AGENT_PID "$SSH_AGENT_PID"
setx SSH_AUTH_SOCK "$SSH_AUTH_SOCK"

(Tip: you can pipe the success messages to /dev/null to not have this pop up on each terminal load)

I then added the following line to the package.json:

"com.company.package": "ssh://git@gitlab.com/company/package.git"

The important bit about gitlab is that their URLs tend to look like this: gitlab.com:company which gives you a hostname resolve error, so be sure to convert that : into a /.

Also: Unity Collaborate ignored the whole package folder which is great, since it previously didn’t when we just had the package with it’s .git folder in there.

Cheers!

maximeb_unity · January 10, 2019, 3:20am

Hi @spaceemotion ,

I’m glad it worked, thanks for letting us know!

Yeah, Gitlab provides SSH “URLs” in the SCP-like format, which is not quite a URL. We don’t support that format (yet) but we intend to because it will be easier - that format is commonly used in the Git world.

Rock360 · January 10, 2019, 1:38pm

I found a good repo!
It works well for me!

https://github.com/mob-sakai/UpmGitExtension

harko12 · February 3, 2019, 7:48am

I think I have set everything up right. The git.exe path is in my System PATH variable. But still every time I try to start up the project, I get:

An error occurred while resolving packages:
Project has invalid dependencies:
com.harkogames.sportfun: No ‘git’ executable was found. Please install Git on your system and restart Unity

A re-import of the project may be required to fix the issue or a manual modification of E:/Unity Projects/MFE/Packages/manifest.json file.

Is there any way I can further troubleshoot why it isn’t finding the git.exe ?

okcompute_unity · February 4, 2019, 1:06pm

Hi @harko12 ,

If you open a terminal, and you type git, does it work?

Also, make sure you close/restart the Hub after installing Git.

Pascal

harko12 · February 6, 2019, 5:37am

I ended up starting unity in Administrator mode, and then it was able to find git. So, maybe it just needed the restart. Thanks!

ShawnFeatherly · February 6, 2019, 8:08pm

I tried to use the file: scheme in Unity 2018.3.0f2. I made a bare git repository (No working directories) locally and can’t get it to be recognized. Here’s the line I’m putting into my Manifest.json "com.frl.versionit": "file:C:\\Test\\VersionIt.git",

The error says “file: Name or service not known fatal: Could not read from remote repository”
Screenshot of error message: https://imgur.com/tCgh0Km

A clone of the repo works fine: "com.frl.versionit": "file:C:\\Test\\VersionIt.gitClone",

Any ideas what I’m doing wrong?

okcompute_unity · February 7, 2019, 12:59pm

Hi @ShawFeatherly,

The local git package scheme you use is not supported for the moment. For the meantime, you need to use this scheme file:///C:\\Test\\VersionIt.git (or file://localhost/C:\\Test\\VersionIt.git if you prefer the explicit form).

Regards,

Pascal

Skjalg · February 7, 2019, 2:08pm

@okcompute_unity I cant get the SCP style pattern to work.

"com.mycompany.library": "git@bitbucket.org:mycompany/my-package.git",

  Project has invalid dependencies:
    my-package: Version 'git@bitbucket.org:mycompany/my-package.git' is invalid. Expected a 'SemVer' compatible value or a value starting with 'file:', 'http:', 'https:', 'git:' or 'ssh:'.

But if I add ssh:// to the path like this:

"com.mycompany.library": "ssh://git@bitbucket.org:mycompany/my-package.git",

Then I at least get a git error

  Project has invalid dependencies:
    my-package: Error when executing git command. ssh: Could not resolve hostname bitbucket.org:mycompany: Name or service not known
    fatal: Could not read from remote repository.
    Please make sure you have the correct access rights
    and the repository exists.

Even though I do have access rights (its a private repo on bitbucket, but i have ssh key for it). I know this because if I open command prompt and type git clone infront of it it works just fine

C:\Users\skjal>git clone git@bitbucket.org:mycompany/my-package.git
Cloning into 'my-package'...

This is tested on Unity 2018.3 and 2019.1.b02. What am I doing wrong? Is it the git url-parsing which is removing the last bit of the url?

edit; It worked when i changed it to git@bitbucket.org/mycompany/my-package.git (replacing the : with /)

okcompute_unity · February 7, 2019, 2:37pm

@Skjalg You hit the other format that we don’t support at this time. For now, SCP and local file: scheme are not supported :(. Sorry about that. As you discovered, in the meantime, you can resort to the SSH scheme URL.

Regards,

Pascal

Skjalg · February 7, 2019, 3:07pm

Yeah thats perfectly fine. What I’d consider a bug though is the : vs / issue in the url

okcompute_unity · February 7, 2019, 5:43pm

The “:” is not valid in an SSH URL. Only with SCP type URL.

keenanwoodall · February 12, 2019, 1:34am

okcompute_unity:

Hi @rz_0lento ,

Glad you have found the pattern :). As stated in the release notes, this is an experimental feature. We are targeting production ready quality for 2019.1. Super glad that you are trying this out though. Please, let us know if you find any issues.

And here is a minimalist documentation if you would like to expand your testing further and maybe for others who are reading this thread.

Requirements

To use Git packages in a project, Git must be installed on the user machine. The Git executable path should be listed in the PATH system environment variable.

Project manifest lock attribute

This feature introduces a new lock attribute in the project manifest. This attribute is populated by the Package Manager to render the package configuration resolution deterministic.

Supported URL schemes/protocols

The Package Manager supports the same URL schemes as Git does. In fact, the URL is passed to the Git executable (minus the revision part). If Git ever adds support for more protocol/schemes, they will automatically be supported by the Package Manager.

List of supported schemes

git:

ssh:

https:

http:

_file: (_only URL-like path are supported for now. i.e. file://absolute/path/repo.git. System path will be supported later. This could allow using relative path for example file:…/…/repo.git)

We also support SCP style pattern:

git@github.com:user/repo.git

Revisions

You can select a specific revision to be cloned by the Package Manager. The revision is prefixed by the # character. The revision should be added after .git extension:

https://repository/path.git#revision

The revision can be any commit-ish - an expression that eventually resolves to a commit hash. The most common types of commit-ish are commit hashes, tags, and branches.

Limitations and good to know:

The Package Manager UI was not updated to support this new package source (ETA 2019.1). I haven’t seen any more issue though.

Big repositories can take a long time to clone. We don’t cache the repository so if you change revision, the repository will be cloned again.

The only way to update a Git package to the latest version is to remove the lock attribute from the project manifest or manually update the revision suffix.

Since the Package Manager uses the Git executable installed on your machine. Any global or system configuration or environment variable will be picked up. These settings may affect the behaviour of the system.

Some package managers offer a shorthand version of their supported URL schemes. For example, npm support short URLs like these: /, gitlab:/. We don’t support these syntaxes for the moment.

Hope this will help you test the feature.

Regards,

Pascal

What’s the plan for package dependencies on assets published to the Asset Store. Will Unity ever support dependencies that reference git repos being published on the Asset Store? I saw you have to have git installed for it to work (makes sense.) Is the final package manager going to ship with some internal version of git, or is using git repos as dependencies only something to be used outside the context of the Asset Store?

keenanwoodall · February 12, 2019, 1:35am

I’d understand if external packages aren’t allowed in stuff published to the asset store b/c you can’t moderate it easily, but it would be nice

maximeb_unity · February 12, 2019, 2:30pm

Hi @keenanwoodall ,

That’s correct. At this time, we have no plans to allow packages to be published with dependencies other than on other packages from the registry. This is both a matter of security and stability.

Due to how Asset Store contents are imported in a project, and the lack of metadata about these contents, it’s currently impossible to know if a given Asset Store asset is present or not, or the version of that asset. The Asset Store package format will in time be migrated to the new format, so that it is possible to reference Asset Store contents from packages.

keenanwoodall · February 12, 2019, 9:17pm

Makes sense. What’s the policy for having dependencies to official Unity packages? What about preview packages? I haven’t seen anything on the Asset Store that depends on Burst or Mathematics yet. I’ve been working on an asset for a long while that has the Mathematics and Burst packages as dependencies. I’m assuming I won’t even have a ability to publish the asset until the packages are out of preview?