We are moving the development of the Graphics packages from the public GitHub to an internal repository while maintaining the public repository as the mirror of our internal development. We are committed to keeping the SRP source open. You can continue browsing this public repository to track the latest changes per landed PR.
Why are we doing this?
We are making this change to improve our developer velocity, decrease the complexity of the development process, and improve our automated testability. This change will allow us to develop features and improve the packages faster, with higher quality and stability. We will also be able to test SRP packages integration better with the other Unity components.
We understand from your previous feedback that keeping the Graphics public repository active and regularly updated is vital for you. We hear that it is essential for you to follow the development progress and have the option to cherry-pick individual changes for your custom forks. Therefore, we are investing in a mirroring service to continuously mirror changes from the internal repository to the existing public Graphics repository. We have looked at different solutions to retain most of the benefits of the current Graphics repo. However, to make sure we can significantly improve the development of the Graphics packages, we have to make some changes.
What will be changed?
Active development: Development activity will move from the public repository to our internal one. The public repository will remain active, and the latest changes will be mirrored to this repository frequently. However, we will mirror the release branches only (2021.2/staging, 2022.1/staging at the moment, and master). We will not be mirroring the development branches.
Change in folder structure: We will move the packages to a new /Packages folder and the content in TestProjects to Tests/SRPTests/Projects. The mirroring service requires aligning the folder structure with the internal unity repository folder structure. We aim to ensure that these changes will automatically merge with your existing local changes. However, you might need to rebase your changes. We are prototyping this now and will provide more info in this thread once we have updates to share.
Changes availability and frequency: The availability of the latest changes will not be immediate. However, the latest changes will be mirrored frequently. Our goal is to ensure that the delay is not longer than three weeks.
Changelogs: The graphics repository packages changelogs will be integrated and synced with Unity Editor changelogs releases. This improvement has already been started with the Unity 2021 releases and ensures that the packagesâ changelogs are automatically available as part of each Unity release note.
Versioning: Going forward, we will increase the minor and patch versions in sync with Unity Editor releases.
The granularity of mirroring: Lowest level of granularity will be a Pull Request (PR). A PR will be squashed and mirrored as a single commit. You will be able to cherry-pick individual PRs for your custom branches.
What will not change?
The SRP and Graphics repository source code remains open, maintained, and updated with the latest changes so you can continue to track the development progress and update your custom forks.
We aim to keep the current package changelogs in the public repository. We are now investigating how to update them from the core Unity release notes automatically.
You can continue working as you are with these public branches. If you have custom branches of SRPs, you can continue to merge and cherry-pick changes as before.
When will the change happen?
We will stop developing in the public repository by February 18. We will then migrate the content over to the internal development repository as the next step. Once all content is in the internal repo, we will run final tests with the new mirroring service. We will then start the mirroring service around March 11.
Thanks for reading. As ever, we appreciate your feedback. You can provide your feedback or post questions in this postâs comment thread, and our team will be on-hand to answer.
I would like to voice my dissatisfaction with this decision.
As many have said before, being able to see the commits is absurdly important, especially with Unityâs track record and even moreso with the Graphics package!
We are all developers here, so we can reason a lot about the development, struggles, goals, etc, just from being able to follow the commits and see the actual progression.
Having access to the code itself is of minimal importance, since we can just take a look at the local package, and decompile any DLLs (as many of us have been doing for years to make sense of Unityâs internals).
Also, letâs not forget that Unity killed the previous GitHub repository that hosted Unityâs decompiled source and then proceeded to ignore the replacement they created themselves for years.
we definitely want our development to be in the open. Every individual PR will be mirrored (as a single commit) so you have more information than just from the source in a released version. This way you can cherry-pick PRs and also get more context about changes in the code. The PR description will be mirrored as well in the commit message.
And although WIP branches will not automatically be visible, our developers will also still be able to share ongoing work in a branch in the public repo to solicit feedback before we land a PR. In the end we want to build tooling around this to easily share WIP branches and integrate this into our process. We are definitely looking into keeping as much of what made the public repo useful for you. We need to start somewhere though and the solution mentioned above is what we know we can do in the short term.
We are definitely committed to keeping the public github and mirroring service going. Itâs likely that other packages that are now developed privately will become public using this approach of developing in the monorepo where everything can be tested together and mirroring out to a public repo to expose a view on the development.
Can you please explain why this is the case? Specifically, how does this move enable you to âdevelop features and improve packages fasterâ?
Without that piece of the explanation, this sort of sounds like a bunch of meaningless marketing speak. I.e. why was the public repository holding your team back in the first place? As a software developer, it is difficult to understand why retargeting your git remote from one address to another would impact development velocity.
This statement is simply incorrect, though, isnât it? Arenât your feature branches being squashed? Cherry-picking changes, then, ends up with a completely different granularity than before. Depending on the number of changes in a given feature branch, it may actually be faster to simply find the relevant lines and copy them around. Whatâs more, individual commits can frequently reveal lots of useful information, especially when the code itself lacks any useful comments. This also requires your teams to have excellent commit discipline, of courseâŚ
Can you please explain why this delay exists in the first place? One would presume that such a system would be 100% automatable. Having a âgoalâ of ânot longer than three weeksâ is far, far less helpful than having a rule. We cannot trust a goal as goals are frequently missed.
Additionally, is that three weeks delay the delay between feature branch merging or actual Editor release? If the latter, then have you considered how this may affect Asset Store Publishers who have to integrate lots of stuff with the SRPs? This would potentially imply that publishers wouldnât have access to changes until âup to three weeks post-new-Editor-release if the goal is metâ. That opens a window where Unityâs and Publishersâ mutual customers will get really upset at Publishers due to assets not being fully compatible on day one (or possibly even month one).
Sadly, thatâs far from enough, especially with the implication that this is not automated.
Iâm sure weâll have many developers voicing similar opinions in this thread, and I hope youâll take those into consideration to, at the very least, keep a full (every commit), automated (and I canât stress this enough) mirror of the monorepo visible to us developers.
Everything is automated indeed but in the first months we are introducing a review stage to verify that the mirroring service does not copy other pieces of the code that could be under NDA. Thatâs why we mentioned the conservative estimate of three weeks. This is the delay between when a PR lands in the private repo and when it is visible in the public repo. This should be visible once the unity editor is released with the change. Once we are confident that the mirroring service is solid and we have the necessary guardrails in place then we can remove the manual verification step to be fully automatic. At that point it should be a matter of days at most.
With this change we are not merely switching the repo from public to internal. We are bringing together the whole graphics stack into one monorepo. The graphics packages and core unity (that contains the platform backends, the graphics abstraction and the core graphics features) will be in the same repo versioned together and tested together automatically. There is a lot of research about the benefits of a monorepo versus a polyrepo. This paper covers some of the pros and cons in large organizations. Every organization is unique though and in Unity developing in one monorepo means weâll be able to have much better automated test coverage of the entire graphics stack, have less complexity in our development process when evolving our graphics stack across the software layers, have faster bisections when investigating bugs on specific platforms, simplify manual QA work, etc. A lot of the reasoning behind why this will help us to accelerate our development is very specific to our current workflows.
Unfortunately, the core unity source code contains confidential platform information so bringing everything together in a monorepo means we need to develop on the internal repo. We definitely did not want to close things up. Thatâs why we are investing in the mirroring service. There are some constrains now currently to make sure we donât mirror other code by accident but once our confidence increases we want to improve on what we presented in our post here.
I always felt Unity should be higher performing, more reliable, cohesive and be somewhere ahead of where it is right now. If this change helps that, then what more could I want?
Provided that the mirroring solution actually works as intended, it seems like we will (mostly) not lose anything aside from increased latency with the ~3 week delay and decreased granularity with the commit squashing. Is my understanding correct?
I maintain our in-house custom SRP and I rely heavily on the GitHub repoâs staging branches for inferences to otherwise undocumented APIs in lieu of any actual documentation (inline or other). If this suffers from this change at all it is taking an already very inconvenient workflow and making it painful.
Could you explain why making these changes does any of these things? What about the current workflow is preventing say automated testing? What workflow issues does using Github have that internal repos donât?
So you cant manage your own development environment without making it private, but expect us to nitpick stuff out of squashed commits? Plus, it is obvious that this "somewhat open code base" idea is not well maintained as it took ages for unity editor csharp code to be updated to the latest editor releases.
The year is 2022. Instead of making the whole engine code public, itâs getting more and more internal and hidden. The granularity of gfx repo was very useful to track individual WIP features and test them on our environment before it lands. So that it is easier to adopt new features and come up with fixes and PRs possibly.
Honestly, I am not happy with this. Why do we have to be constantly disappointed in using Unity instead of Unreal?
[quote=âLordDarkon76, post:12, topic: 871814, username:LordDarkon76â]
Git has the option for Submodules it isnât a technical problem to have the module public.
[/quote]We actually donât use Git for the core Unity source code today (though weâre working on transitioning to it this year), but submodules also do not eliminate the need for multiple pull requests in separate repositories. By moving to a more monorepo-like setup, graphics developers will be able to land core-side and SRP-side changes atomically.
As @AljoshaD mentioned, a major concern is NDA materials - e.g. we cannot store CI job definitions for NDA platforms in the public repository like we do for other platforms, because it would disclose things like the names of executables in the SDKs (and we could debate whether itâs really necessary for NDAs to protect such things, but nevertheless, they do protect such things, so we honour that). Weâve done our best to work around the problem from early in the SRP project, but the result has been a huge amount of friction for the developers. Moving the âsource of truthâ to be internal allows us to incorporate all of these NDA pieces directly into the repository, just like any other platform, and then just strip them out in our mirroring process, much like we already do for customers who license the core Unity source code.
We know that being able to see and understand the history of development is valuable information, which is why as @ali_mohebali said, weâre investing in a mirror service - we want to mirror as much as we can*,* while still protecting our partnersâ IP, and without creating lots of friction for developers. As @AljoshaD said, âthree weeksâ delay is a conservative estimate while we get confident that the process is working correctly; we absolutely want to get to a place where we are much faster than that.
Many of the existing PRs/commits targeting stable versions of HDRP are themselves already a big group of sometimes unrelated changes - for example this one https://github.com/Unity-Technologies/Graphics/commit/3179cc5e90c0bc23f00d0489f1f48f664f13614a . With the current setup itâs already kind of annoying to trace actual changes through those monolithic commits (doing so has been necessary many times for me as part of making my large Unity project shippable as well as for submitting bug reports that are actionable by Unity), but since those monolithic commits link back to individual smaller PRs at least Iâm able to do that tracing myself.
It sounds like the new setup might lose useful information like that (e.g. PR links or commit hash information for the original changes a monolithic backport PR is comprised of) â is that an accurate guess, or do you have a plan to make those at least as usable as they are currently?
Also, will existing/old PR and branch information be preserved once the mirroring begins? Losing the old ScriptableRenderPipeline repository a couple years ago meant a bunch of history links got broken and it got harder to debug issues, Iâm hoping something similar doesnât happen again.
Unfortunately it sounds like the development of the Unity projects I work on will be hurt by this change because at least some of the workflow points I brought up the last time around will still be impacted: https://discussions.unity.com/t/822248 page-3#post-6699961
For example Iâve been keeping a close eye on the HDRP water system pull request content and comments (not just commits) recently and itâs helped guide where I spend time, and now Iâll be losing some of that.
In the new setup all these will land in separate PRs instead of a batched PR. This will make it much easier to cherry-pick at that granularity.
yes it will be preserved. We will link up the history. The aim is that you can just continue pulling in changes from the release branches. However, we need to change the folder layout at the same time. Weâll be running experiments in the next weeks to see how well git can handle our mirror setup and what steps are needed to rebase an existing fork.
Also, all existing PRs, branches and history will remain available in the Graphics repo.
