is there a way to have a standalone c# exe launch a unity game not as a separate process but within itself?
No - Unity is an application done mainly in c++, can’t work really, it’s not an sdk alone
If you turn your problem on its head there may be a solution – instead of embedding a Unity game inside your standalone C# app, you could embed your app inside a Unity game. You can build a standalone executable using Unity, and you can use Mono to integrate pretty much any .NET DLL you want, including those you’ve built externally. You’d have to jump through some hoops to get your Unity game to appear in a window inside your app though, not sure how that would work.
Alternatively, you could build your Unity game as a web player, and then embed a web browser window inside your app and host the Unity game there. This might be a better solution.
Let us know if it works, it’s an interesting challenge.