Code Mode for Unity Editor - advanced way for LLM agents to work in Unity Editor environment

[roma_rogov]

I would like to introduce you my approach in communication between AI agents and Unity Editor - Code Mode for Unity, which is completely free for now and represents my research results in agentic LLM usage in game development.

GitHub repository

Sorry, no ads-like cool video with claude building simple scene from primitives for you today – I have no time for video editing and my goal is to build real tool for real projects, not yet another MCP toy. You can easily try it yourself in a few simple steps, though.

What is Code Mode?

In contrast to rigid MCP tool defenitions, which always kept in LLM context, CodeMode is an approach which helps AI to call tools in the most familiar way - by writing JavaScript code based on TypeScript defenitions of tools. This helps AI to keep token consumption low, implement loops and chained calls for complex tasks, organize output in compact form and reuse output from different existing servers and endpoints in one JavaScript execution context, isolating LLM context from unnecessary data.

How it can be used with Unity Editor?

Code Mode for Unity runs an HTTP server inside the editor that exposes scene manipulation, asset management and property inspection as structured tool calls via UTCP Protocol - letting AI agents build, inspect, and modify Unity projects the same way a developer would through the UI.

These tools are combined in UTCP Code Mode environment to achieve maximum performance and token efficiency for AI agents, letting them call the tools in isolated JS sandbox.

Flexible tools architecture

Tools exposed in Code Mode are arranged in clear, straightforward and domain-specific manner, allowing LLM to easily select specific tools for exact task. On development of this toolset I was inspired with actual Unity Editor UI, because this is real, existing and long-tested user experience which helps millions of users get things done every day. Here is quick overview of tool categories and how they are related to Editor’s UI:

Category	Tools	Purpose
Scene	GameObjectGetTree, GameObjectGetAtPath, GameObjectCreate, GameObjectCreatePrimitive, GameObjectOperate	Navigate and build scene hierarchy
Components	GameObjectComponentsGet, GameObjectComponentAdd, GameObjectComponentRemove, GameObjectGetAvailableComponentTypes	Attach, remove, and discover components
Inspector	InspectorGetInstanceDefinition, InspectorGetInstanceProperties, InspectorSetInstanceProperties	Introspect types and read/write properties
Assets	AssetGetTree, AssetGetAtPath, AssetCreate, AssetImport, AssetOperate, AssetGetPreview	Browse, create, and manage project assets
Editor	EditorOperate, EditorGetLogs, EditorGetScenePreview	Control editor state and capture previews
Settings	SettingsGetDefinition, SettingsGetProperties, SettingsSetProperties	Read and modify project settings

Current goals

This is an early release, which already helped me in my development process, and for now I just want to share it with the community and see how much interest this approach can get. Personally I see a lot of potential in it, and it would be great to hear your thoughts and maybe even see if it helps you in integration of AI in your development pipeline.

Please, read more at github page, test this thing out, look at the actual implementation, write feedback and reviews - I would really appreciate any sort of contribution!

You can always ask me a question or two by DM in LinkedIn

Have a nice day!

[hiptopjones]

Cool project! I think there’s a lot of potential here. It’s great to see different ways of enabling LLMs to work with Unity.

I got it working in Claude Code, and was able to get some changes into the editor. It was often hitting the same types of failures, mostly because it was trying to write code without querying for tools, so the system prompt probably needs work. I’m going to keep messing with it.

I think even a simple video of setup and demo interaction would go a long way to encouraging more people to check it out.

BTW, I upgraded your project to Unity 6.3 and found that the server would hang on the Repaint() call in UtcpServerWindow.OnServerLog(). I was able to just comment it out for my experiment, but something to look at.

Curious to see where this goes!

[roma_rogov]

Thank you!
It’s very helpful to see such kind of support.

Yes, it’s pretty raw implementation for now, and this approach requires more fine tuning in prompt, I’m working on it. Actually I’m usually start with generating custom prompt and skills in Claude depending on relevant action area. In the future I’m planning to move sandbox from JavaScript to dotnet/Roslyn implementation, but going to keep sandbox code execution isolated from Unity API for safety. I’m already collaborating a bit with UTCP team in their CodeMode development, so I think C# implementation would be welcoming from their side
Also I’m thinking about moving tools implementation to project instead of readonly package, so developers or AI can adapt and improve them with strict developer review, yet my main goal is to keep editor context safe.

As for video - I just want to make concept explanatory, since it’s not simple architecture to catch quick attention, and simply don’t have enough time at the moment because looking for a new job place

[hiptopjones]

@roma_rogov

In the future I’m planning to move sandbox from JavaScript to dotnet/Roslyn implementation

Yeah, was thinking that it would be nice to see C# rather than TypeScript. Something like DotNetIsolator might also help with building out a custom execution environment.