Blog Post: Introducing Unity Muse and Unity Sentis, AI-powered creativity
There were some teasers about this in the past but now there’s some actual footage of the tools. Looks interesting but doesn’t seem to do anything different than existing popular AI tools out there. Does anyone already use this and have more details?
I haven’t seen inpainting for sprites with transparent backgrounds or transparent background sprite generation in general. But I also haven’t really looked for any AI tools like that. Maybe it already exists. Seems like a cool way to prototype assuming it’s not prohibitively priced. CEO strongly nodded that they’ll be priced by usage according to the earnings call earlier this year.
I’m sure the devs worked very hard on these tools, and I don’t want to shit on them, but the results don’t look great. Should these terrain textures that look like they were pulled from a 90s game be what they showcase?
The talking alien showcase felt kinda lifeless. If I were to make a realistic-looking game in Unity I wouldn’t settle for the quality of content that was generated by Sentis.
Yeah exactly, that’s why I wonder if anyone here has used it to give some info, but they are probably not allowed to talk about their experiences in the closed beta yet.
For example, the “chat” demo they have looks similar to ChatGPT (except trained on up to date unity docs and resources) but in terms of productivity I’ve found AI Code Tools such as Github Copilot (or Copilot-X) to be much more useful. Being able to ask specific questions about Unity and documentation certainly is useful though but that doesn’t feel viable as a paid product (which I assume this will be)
In regards to art generation, Stable Diffusion is really powerful once you get into all the extensions and tools out there. What was shown in the demo seems very basic and unless they do a lot of prompt juicing behind the scenes to help users get what they want I think it will be just giving us those stale looking sprites and textures. MidJourney for example does so much prompt tuning behind the scenes that even really simple prompts from users can generate “good” looking images. But if you take a basic generator without any tuning then it gives images like in the unity demo (looks like something you can make in paint)
There’s some game specific tooling out there such as https://layer.ai/
They are more tailored for game assets as it can do transparency, pixel perfect, etc. It supports generating assets from traces/sketches and also has in-painting directly in the tool.
their demo (couldn’t figure out how to embed it):
Edit: just noticed unity lists Layer.ai as a solutions partner in their blog post
Isn’t this just post IPO Unity jumping on the next biggest bandwagon?
Muse doesn’t even seem to be anywhere near where it’s at to ask ChatGPT for Unity assistance.
As others mentioned, image generation looks bad.
The poor quality of the textures may be due to the SD 1.5, which creates 512x512 images. Additionally, the generator is limited to creating only albedo textures.
Inpainting ignores the original style of the sprite, they haven’t implemented Controlnet into their product.
We also don’t know all the quirks this tool might require. With SD, you need to learn artists and how to compose them to get a good style.
I’m not impressed, feels like a rushed product.
Unity invested in AI developmet years before recent LLM hype, but I’m not sure if any of this is related to that.
And by “invested” we really mean “bought companies that looked shiny”. 
As far as I’m aware, Barracuda was developed in-house: https://github.com/Unity-Technologies/barracuda-release Earliest public release is from 4 years ago when no one gave a shit about AI.
The question is if Barracuda is at all relevant at this point.
EDIT: ChatGPT4’s opinion after searching the web via plugin:
"Unity Barracuda, as of 2023, seems to remain relevant in the context of large language models (LLMs) like ChatGPT, particularly for the gaming industry. Unity is focusing on deploying generative AI in games, which they believe will drive a significant transformation in the gaming experience1. With Project Barracuda, Unity aims to enable AI models to run locally on a device, which could help scale AI character implementation in games without incurring substantial costs1.
The use of generative AI, such as LLMs, in game development has the potential to enhance the richness and believability of non-player characters (NPCs) and the overall gaming world. This could result in unique game experiences with persistent and evolving contexts and characters for each player1. Unity’s CEO, John Riccitiello, believes that game designers and developers could be up to 10 times more productive using AI, and while AI won’t take anyone’s job, those who utilize AI in their work may outperform those who don’t1.
However, it’s important to note that the capabilities of LLMs in game creation are not without their limitations. While they may eventually produce simple games, creating complex and compelling games is likely to remain a challenge. This is due to the dynamic nature of games, where interactions between characters can change the gameplay in real time, something that is fundamentally different from the types of content LLMs typically study and learn from1.
In parallel, advancements in integrating LLMs with development tools are evident. For instance, JetBrains has introduced AI features powered by LLMs in its Rider IDE for .NET development. These features allow developers to interact with the LLM to ask questions or iterate on a task, generate commit messages, and even generate files for a Unity solution written to developers’ specifications2.
Given the rapid advancement of AI, it’s challenging to predict its future capabilities. If AI continues to be integrated into game creation and operation engines, it might be capable of learning from the millions of games it helps create and operate, potentially leading to further advancements and improvements1. It’s also worth considering the potential economic impact. If generative AI facilitates significant growth in the gaming industry, it could represent a substantial proportion of global value creation in the coming years1.
The more that’s revealed about this AI stuff, the more it just looks like they plugged some ChatGPT outputs into a few basic inputs for some of the most mundane aspects of game development. We’ll be able to set some color inputs, and get some bizzare AI generated character dialogue that’s completely isolated from in game mechanics. A lot of fuss to integrate a bit of ChatGPT into Unity and marry the usual overhyped Houdini style modularity with some very narrow AI based points of input kneecapping both systems that already show little promise in creating great games.
Nothing here looks groundbreaking in the slightest.
I’m just hoping that in cleaning up the code to be utilized by AI, they inadvertently make it easier for users to work with the various systems.
Dare we dream…
I’m more worried they’ll come to rely on the ChatGPT client they host in the Editor to save them from ever writing more documentation, or editing it to improve reference materials and better explain feature functionality and mechanisms. etc.
ChatGPT can’t produce animation. Stable diffusion out of the box doesn’t really produce materials.
Alot to unpack from the video!
Some interesting tools there; if the data sets trained on are legal and ethical and the outputs are decent-enough I can see myself using AI Animation generator alot for placeholders and blend fills and also the sprite generator/variant creator for also placeholders and background art. Would love to get on the beta to try them out. 
What I find hilarious though is that the example for Unity Muse for ‘How do I make a match-3 game’ is exactly how you should NOT make a match-3 game; the combination of 2D Rigidbodies on cell tiles of the Tilemap system doesn’t make any technical sense. This part was definitely not fact-checked by anyone who has used these Unity tools. 
I asked the same question to ChatGPT and it gave me a vague but fairly solid outline which is more aligned with reality and doesn’t set you down a path of frustrations.
Also, the Unity 3.0 era terrain demo for ‘Realistic Textures’… Big yikes.
The marketing materials state they won’t generate entire materials but textures to use in unity.
Second thing, base stable diffusion can generate textures. https://pixela.ai/
Shows it might be true, their claim that Muse is trained on the Unity documentation. The samples and examples throughout the Unity docs always use the least efficient way of doing anything. As did Brackey’s vids.
I don’t think its fair to mention Brackeys; he was an independent youtuber and his videos were bitesize focuses on beginner & intro topics, and weren’t advertised as ‘most efficient’. For what his channel aimed to do and who it was for; hit the mark perfectly.
Unity is a public corp with 5k employees and $billions so has a duty, scale and resources to provide the best standards and workflows for creating something with it.
Aren’t they using Barium AI for pbr texture generation which they acquired earlier this year?
Barium AI is based on stable diffusion with some custom scripts and obviously a custom model.
https://discussions.unity.com/t/907606/6
We’re each responsible for the lies we tell and the truths we hide.
#deep
