[Open Source] UnityNeuroSpeech – Real-time voice AI in Unity

[HardCodeDev777]

Make your Unity characters hear, think, and talk — using real voice AI. Locally. No cloud.

---

UnityNeuroSpeech is an open-source framework for creating fully voice-interactive AI agents inside Unity.

It connects:

• Whisper (STT) – converts your speech into text
• Ollama (LLM) – generates smart responses
• XTTS (TTS) – speaks back with custom voice + emotions

All locally. All offline.
No subscriptions, no accounts, no OpenAI API keys.

---

What can you build with UnityNeuroSpeech?

• AI characters that understand your voice and reply in real time
• NPCs with personality and memory
• Experiments in AI conversation and narrative design
• Voice-driven gameplay mechanics
• Interactive bots with humanlike voice responses

---

Core Features

Feature	Description
Voice Input	Uses whisper.unity for accurate speech-to-text
AI Brain (LLM)	Easily connect to any local model via Ollama
Custom TTS	Supports any voice with Coqui XTTS
Emotions	Emotion tags (<happy>, <sad>, etc.) parsed automatically from LLM
Actions	Action tags (<turn_off_lights>, <play_cutscene_123>, etc.) also parsed automatically from LLM
Agent API	Subscribe to events like BeforeTTS() to monitor your agents
History Saving in JSON	Save dialog history between player and LLM in JSON with/without AES encryption
Editor Tools	Create, manage and customize agents inside Unity Editor easily
No Cloud	All models and voice run locally on your machine
Multilingual	Works with over 15+ languages, including English, Russian, Chinese, etc.
Multiple Voices and Languages for Multiple Agents	Each Agent can have each voice file for any available laguage
High Performance	Uses UniTask instead of Coroutines and Tasks for optimal performance.
Full Build Support	Full compatibility with both Mono and IL2CPP scripting backends.

---

Built with:

• OllamaSharp (Ollama)
• whisper.unity
• UniTask
• Coqui XTTS
• UV
• Unity 6

---

Repository - GitHub - HardCodeDev777/UnityNeuroSpeech: The world’s first game framework that lets you talk to AI in real time — locally. Supports any custom voice.
Documentaion - UnityNeuroSpeech Docs

Any feedback, issues, or contributions are welcome. Thanks for checking it out!

[DigitalAdam]

Under Microphone dropdown: What goes there? I can’t create the manager unless I fill that out.

[kieec]

You don’t have to do anything to this dropdown. This dropdown is for assigning microphone list after Unity playing. By the way, If you successfully run this project, see if you can get emotional tag from AI response. I tried for couple of weeks and still can’t get emotional tags like happy. sad,…etc.

[HardCodeDev777]

Hi! Emotional and actions tags aren’t hardcoded - everything depends on LLM. UNS only gives strict system prompt to them - and sometimes it may not work. More information will be helpful - model, logs and maybe even generated ScriptableObject with Agent settings. But I would recommended you to make an issue in UNS repository(btw here I solved DigitalAdam’s problem).

[breadandbutter1999]

i readed in their notes that the emotional tag is less responsive on smaller models, i will have to test the out but they say to use a large model for that

[breadandbutter1999]

i get an error about a switch2 not a valid name in Unity 6. i was able to delete it and fix the issue

[breadandbutter1999]

Does anyone have a more detailed guide?

[lettheweedbun420]

Hi HardCodeDev777. i just try your code i got this error “[UnityNeuroSpeech] Agent 0 TTS error(or warning): ‘tts’ is not recognized as an internal or external command,”

. How to fix this

[unity_78AB672B3D86B175390C]

I found the solution for that issue. Replace the tts fileName with absolute file path"../ local/bin/tts.exe" and remove the string from argument “/C tts” after hit play button it will work properly.

[HardCodeDev777]

Hi! This error means that TTS wasn’t added in PATH enviroment. This is very weird if you used setup.bat for installation.

Solution about adding absolute file path on your pc is incorrect, it just won’t work for players. I’d recommend you to run uv tool install coqui-tts and then show result(cmd messages).

[HardCodeDev777]

I’ve never seen this error. Could you provide errors and other messages in Unity console?

[kieec]

At last, I use Ollama llama3.1 model with a ModelFile pasted your strict system prompt to create a custom llama3.1 model. And now it can show emotion tag at each time replying.

[Selecur]

Hi. When you use the .bat, it says uv and coqui was successfully installed, but when checking the logs, it says failed to install due to llvmlite

The error seems to happen because system couldn’t find python

[HardCodeDev777]

I didn’t add any validation in setup.bat, this is why I provided logs. Kinda bad way to make setup tbh. But I’ll fix it in 2.1.0(coming very very soon btw).

And I think I found the solution for Python error:
If you stil have uv installed, run uv tool install --python 3.11 coqui-tts. If it helps, then the problem was in Python version.

EDIT: The problem was in Python version - I fixed it. Also I released 2.1.0 with validation in setup

[Selecur]

Nice! I’ll check it out

[y0_0go]

I even got Ollama response text. But after that, in the TTS process, this error occured @HardCodeDev777

[HardCodeDev777]

When I released the latest version of UnityNeuroSpeech(2.1.0), the latest version for Coqui TTS was 0.27.2 and everything worked correctly.

But recently Coqui TTS released 0.27.3 and, as you can see, it’s completely broken. But ofc UV will install the latest version. I actually thought that they won’t release broken versions.

So, now you just need to reinstall it with 0.27.2 version.

I hope it’ll help:

uv tool uninstall coqui-tts
uv cache clean
uv tool install --python 3.11 "coqui-tts==0.27.2" 

Write it in cmd.