I tried to properly fix it. I debugged the binary, I patched managed memory and I got very close but still too far to call it a working solution.
I tried all my gamedev contacts that could have access to Unity’s source code, but nobody does.
I’ll leave here my observations, maybe somebody more skilled in reverse engineering than me could use it. But take everything with a grain of salt.
Unity uses GDK/GTK for native UI rendering, and as you know it doesn’t have fractional scaling. Unity just doesn’t know your monitor is scaled. But it has a method called GetGtkWindowScale that wraps gtk_widget_get_scale_factor(GtkWidget*) and casts it returned value to float (thanks dude, much appreciated). It also fallbacks to the default value, that is set in method InitializeGtk.
I tried forcing this value to 1.5f, and it works - ui scales to 150%, but it also goes beyond panels (rendered as GtkDrawingArea).
Unity stores UI elements’ coordinates in logical points that later translate into viewport points and then into physical ones. Apparently, this GetGtkWindowScale method is used in too many places and viewport size needs to be compensated. I tried all of that, of course and it didn’t work for me, but probably I did something wrong.
I also tried countless places in this UI rendering chain: GUIView::SetPosition, GUIView::UpdateScreenManager, some others that I can’t remember now.
Unity UI isn’t very consistent, it looks like there are still some panels that weren’t migrated to UI Toolkit. It makes debugging even more complicated, they don’t respond to patches the same way.
So debugging left me with a feeling that every my patch was either miss or too broad - I couldn’t patch exactly viewport size calculation.
Then I tried to check managed memory. Likely, we have source code of it and it’s much easier to patch.
Apparently, scaling is Panel responsibility. So I tried patching editor panels to have bigger scale and it works! There are some problems though: Mouse position needs to be compensated. I managed to find some places where events are processed, but scene view didn’t work. And popups were cropped and placed wrongly too. Oh, and images scaled pixelated.
Proper way would be to force Panel.pixelsPerPoint, but it looks like this patch would require patching both managed and unmanaged memory.
And I want to mention that I decided to switch from Windows to Linux because all tools I need are supported, most importantly, Unity. Well, this wasn’t the warmest welcome.
I think at this point I don’t want to spend countless hours poking the debugger, I’m tried.
@Tautvydas-Zilys if you could help with just explaining why it doesn’t work, what needs to be fixed, I’ll try to do that. There is a very old issue that apparently not going to be fixed anytime soon: