UnityAI V2 – Autonomous Unity Development Agent
UnityAI V2 is a modular autonomous development agent for Unity. Its goal is to plan, implement, validate, repair, and continue development tasks across code, scenes, prefabs, runtime behavior, Blender, and locally generated 3D assets.
The system is built around a backend-agnostic planning architecture:
Planner
↓
Tool Plan
↓
Validation
↓
Approval
↓
Coordinator
↓
Executor
↓
Unity / Blender / External Backends
Core Development Pipeline
UnityAI V2 already supports:
-
Development plans and planning sessions
-
Step-based execution
-
Waiting states and resumable workflows
-
Compile tracking
-
Validation tracking
-
Patch generation, preview, and application
-
Automatic compile repair
-
Approval policies
-
Developer sessions
-
Automatic replanning after failures
The code pipeline works like this:
Patch Generation
↓
Preview
↓
Approval
↓
Apply
↓
Compile
↓
Validate
↓
Automatic Repair
↓
Continue
Generic Tool Framework
UnityAI V2 is not limited to editing source code. It also has a generic tool system for structured Unity operations.
Implemented tool areas include:
-
Scene tools
-
Prefab tools
-
Serialized property tools
-
Runtime tools
-
Play Mode validation
-
Asset transactions
-
Stable object references
-
Structured post-validation
-
Automatic tool repair and replanning
-
End-to-end tool tests
The system uses a hybrid execution model:
Patch Steps
↓
Compile
↓
Repair
Tool Steps
↓
Unity
↓
Validation
↓
Repair
This allows the agent to combine source-code changes with direct Unity Editor operations in the same development plan.
Blender Integration
Blender is integrated as a separate backend rather than being embedded directly into Unity.
Development Agent
├── Unity Backend
├── Blender Backend
├── Git Backend
├── Build Backend
└── Validation Backend
The Blender pipeline currently supports:
-
Blender connection and job execution
-
Mesh generation and import
-
Existing model import
-
Mesh cleanup
-
UV generation
-
Material processing
-
Rigging
-
Skinning
-
Animation clips
-
Unity Humanoid setup
-
Animator runtime validation
A typical asset pipeline looks like this:
Generate or Import Mesh
↓
Cleanup
↓
UV and Materials
↓
Rig
↓
Animations
↓
Export
↓
Unity Import
↓
Prefab Creation
↓
Runtime Validation
Local AI 3D Generation with ComfyUI
UnityAI V2 is now being extended with a local ComfyUI backend.
The current ComfyUI workflow can generate a concept image from a text prompt and then convert that image into a textured 3D model locally. The complete prompt-to-model process currently takes around 30 minutes.
The planned integration is:
UnityAI
↓
ComfyUI.Generate3DAsset
↓
ComfyUI REST API
↓
Prompt → Image → 3D Model → Textures → GLB
↓
Temporary Asset Staging
↓
Blender Cleanup and Processing
↓
Unity Import
↓
Prefab Creation
↓
Play Mode Validation
UnityAI submits an exported ComfyUI API workflow through POST /prompt, receives a prompt_id, stores the job persistently, monitors the job through the ComfyUI history API or WebSocket connection, and locates the resulting GLB file.
Generated files are not written directly into the Unity Assets directory. They first go through a staging area:
ComfyUI Output
↓
Library/UnityAI/GeneratedAssets
↓
Validation
↓
Blender Processing
↓
Asset Transaction
↓
Assets/UnityAI/Generated
This prevents incomplete or invalid files from triggering Unity imports.
Persistent Long-Running Jobs
Because local 3D generation can take around 30 minutes, generation jobs are stored persistently.
Example states:
Created
Submitted
Queued
Running
Completed
Output Located
Staged
Blender Processing
Importing
Validating
Succeeded
Failed
Cancelled
This allows UnityAI to recover after:
-
Unity domain reloads
-
Script recompilation
-
Editor restarts
-
ComfyUI disconnections
-
Blender failures
-
Import failures
An important design rule is that later failures do not restart expensive generation steps.
For example:
ComfyUI Generation: Successful
Blender Cleanup: Failed
The repair system retries the Blender step using the existing generated GLB instead of running another 30-minute generation job.
Workflow-Based Generator Abstraction
The Planner does not need to know whether a workflow internally uses Stable Fast 3D, TripoSR, Hunyuan3D, Flux, or another model.
It only sees a generic tool:
ComfyUI.Generate3DAsset
The request may contain:
{
"prompt": "A stylized medieval treasure chest",
"workflowId": "prompt-to-3d-v1",
"assetName": "MedievalTreasureChest",
"seed": 123456,
"runBlenderCleanup": true,
"createPrefab": true
}
Provider-specific node IDs and workflow details remain inside a versioned workflow definition.
This means new local models can be added by registering another ComfyUI workflow instead of modifying the UnityAI planner.
Safety and Approval System
UnityAI V2 uses scoped permissions and approval policies.
Different backends are allowed to modify different locations:
ComfyUI Backend
→ External or temporary output directories
Blender Backend
→ Temporary processing directories
Unity Asset Backend
→ Assets/UnityAI/Generated/**
Patch Backend
→ Approved source-code locations
Operations can be classified by risk, for example:
-
Compute intensive
-
Project mutation
-
Destructive
-
External service
-
Asset overwrite
-
Runtime execution
This allows approval policies to be based on actual risk instead of a simple destructive/non-destructive flag.
Completed Major Milestones
The project has completed the following major areas:
-
AI foundation
-
Development planning
-
Patch pipeline
-
Autonomous code agent
-
Workflow refactoring
-
Approval and safety
-
Testing infrastructure
-
Generic tool framework
-
Tool validation
-
Scene tools
-
Serialized property tools
-
Tool end-to-end tests
-
Prefab tools
-
Stable references
-
Asset transactions
-
Structured post-validation
-
Runtime and Play Mode tools
-
Automatic repair and replanning
-
Hybrid code and tool agent
-
Blender integration
-
Rigging, animation, Humanoid, and Animator validation
Current Development Focus
The current focus is the ComfyUI asset-generation backend:
22A – ComfyUI Backend
- REST client
- Workflow registry
- Dynamic prompt and seed injection
- Persistent generation jobs
- Output discovery
- Asset staging
- Artifact registration
22B – Live Monitoring
- WebSocket integration
- Queue status
- Node-based progress
- Cancellation
- Editor UI
22C – Full Pipeline Integration
- ComfyUI
- Blender
- Unity import
- Prefab generation
- Runtime validation
- Visual quality checks
- Automatic repair
Long-Term Goal
The long-term target is a complete autonomous asset and gameplay development pipeline:
Specification
→ Generate
→ Cleanup
→ UV
→ Materials
→ Rig
→ Animation
→ LOD
→ Unity Import
→ Avatar
→ Animator
→ Prefab
→ Play Mode
→ Visual QA
→ Automatic Repair
The main architectural principle is that the Planner remains independent from individual tools and providers. Unity, Blender, ComfyUI, source-code patching, build systems, and future services are all treated as interchangeable execution backends behind validated tool contracts.
UnityAI V2 is therefore evolving from an AI code assistant into a resumable, repair-capable autonomous Unity development system.
=============================================================
What i really want to say is This hole Content can be used with a Cloud AI like OpenAI or Mistral AI, or you can use your local Maschine like Ollama or LM studio to get a working Plan …
=============================================================
@ Last here is a Prompt i working today with UnityAIv2 successful with it:
Generate the complete animation pipeline for the existing calibrated character model.
Character.DefineSpecification must use:
Version: 1.0
Name: MeinModel
RigType: Generic
Height: 1.8
Animations: [Idle, Walk, Run, Attack, Death]
GeneratePrefab: true
Use the following as the unmodified source model:
Assets/UnityAI/Generated/Models/MeinModel.glb
Blender.RigAndAnimateModel must use:
RigPreset: GenericRootMotion
RigCalibrationPath:
Assets/UnityAI/Generated/RigCalibrations/MeinModel.rig.json
ClipName: Idle
ClipNames: [Idle, Walk, Run, Attack, Death]
Do not invent markers and do not use automatic bounds rigging.
Fail if the calibration file cannot be loaded.
Save the fully animated FBX at:
Assets/UnityAI/Generated/Models/MeinModel_Calibrated_Animated.fbx
Subsequently, import the FBX into Unity and configure it as a Generic Rig.
Import all included animation clips.
Configure and validate the clips as follows:
Idle:
- Loop: true
- IsRootMotion: false
- ExpectedDirection for RootMotionVariants: None
Walk:
- Loop: true
- IsRootMotion: true
- ExpectedDirection for RootMotionVariants: Forward
Run:
- Loop: true
- IsRootMotion: true
- ExpectedDirection for RootMotionVariants: Forward
Attack:
- Loop: false
- IsRootMotion: false
Death:
- Loop: false
- IsRootMotion: false
ExpectedDirection must be passed exclusively to Unity.BuildRootMotionVariants
and never to Unity.ValidateAnimationQuality.
Analyze the skeleton and validate all five animations using
Unity.ValidateAnimationQuality. Create an Animator Controller at:
Assets/UnityAI/Generated/Animations/MeinModel_Animated.controller
The controller must contain these states:
Idle
Walk
Run
Attack
Death
Idle is the default state.
Use the following parameters:
Speed (Float)
Attack (Trigger)
Die (Trigger)
Create the following transitions:
Idle -> Walk, when Speed is greater than 0.1
Walk -> Run, when Speed is greater than 0.65
Run -> Walk, when Speed is less than 0.65
Walk -> Idle, when Speed is less than 0.1
Any State -> Attack via the Attack trigger
Any State -> Death via the Die trigger
Finally, create a prefab at:
Assets/UnityAI/Generated/Prefabs/MeinModel_Animated.prefab
Assign the generated controller and the generic avatar to the Animator.
Set Apply Root Motion to true.
Do not overwrite any existing FBX, controller, animation,
or prefab files.
If you are intrested i can show you my results of it …
