Introducing Unity Muse and Unity Sentis, AI-powered creativity

Today we’re announcing two new AI products: Unity Muse, an expansive platform for AI-driven assistance during creation, and Unity Sentis, which enables you to embed neural networks in your builds, enabling previously unimaginable real-time experiences.

Whether you’re just getting started or already a Unity expert, Unity Muse and Unity Sentis are here to help you open up new avenues of creativity and innovation. We’re committed to working closely with you to explore, validate, and define our AI offerings, which is why both will be available via closed beta at this time.

Read our blog post for more information

13 Likes

This sounds really awesome

I’m just wondering though since its going to be a closed beta, will there be issues if I use the tools for the game I’m currently developing when muse AI starts out.

Other than that I can’t wait for it

Sounds good. Really gamechanger. But we can talk about it only after release.

This should be very interesting. I look forward to seeing more. What makes this special isn’t really the AI, but the Unity ecosystem working with it to include it in workflows. Hopefully this will improve productivity!

1 Like

Since the beginning of the year I was thinking of using something similar to GPT for my open world game, but the good solutions ran in the cloud and there were API costs, not good thing for immersive games. Unity Sentis is exactly what I was waiting for!

I hope it has some good and easy training, something like Character.AI.

3 Likes

I’m all signed up! I’m in the beginning prototype phase for a casual family-friendly VR multiplayer game I’m making by myself and launching in 2 years with many more years of updates.

2 Likes

Can we find somewhere on what data Muse was trained? Or at least get the guarantee it’s trained on legally unproblematic data? With how the AI landscape is rapidly evolving, and laws will likely follow with some delay, it would be good to know any generated data is always going to be in the clear if we end up using it in our applications.

14 Likes

I’m assuming that the alien looking character from the Unity Sentis demo is an AI 2D animated model… similar to D-ID generated content, is that right or?

Great question! So, you cannot use it for a current commercial or external project. The beta is more meant to help test out and identify any issues.

Very cool. I signed up a while ago, I want to play with anything AI related

1 Like

To create Unity Muse Chat, we licensed third party LLMs and integrated them with first-party Unity technical documentation (including manuals, release notes, tutorials, scripting guides, etc.) so creators can get the most relevant and up-to-date information.

2 Likes

I came to hear about lots of concerns regarding the contents generated by AI being used in game development, do we need to provide addtional layer of licensing or anything of that sort if we are using sentis or muse for generating in game contents like materials or characters.

3 Likes

In the video only 2D sprite, texture and animation generations are mentioned. I have couple questions.

  • Will the animation generation only restricted for humanoid animations or will it include generic animation ?
  • Do you also plan to work on model generation, particles, shaders etc. ?
  • What is the underlying technology? GPT-4 ?
2 Likes

I would assume that applies to the chat-based app/game development instructions. Does this extend to the other functionality you show such as character animation as well as the sprite and texture generation? Is that all based on first-party Unity content or for-purpose source assets from other partners?

2 Likes

Great job Unity! I’m so excited for Sentis to use with NPC conversations!

2 Likes

Which tool has been used for the facial animation of the alien head and does it automatically generate exact phoneme to viseme mapping?

1 Like

Okay, but how does this answer square with your commitments to be transparent? This is from your AI guidelines:

You are being Opaque while pretending to be Transparent.
For Clarity:

  • What Models are you using?
  • What datasets were THEY trained on?

If you only know the answer to Question 1, that’s STILL useful.

Until then, I’m just going to assume that

  • Muse is a ChatGPT wrapper
  • your facial animation system is just an Nvidia Omniverse Audio2Face wrapper
  • Your “Texture Generator” or “inpainting tool” is just a Stable Diffusion wrapper

Which means they’re all trained on dirty datasets containing millions of Copyright Infringing works.

17 Likes

Hi. What’s the difference between Barracuda and Sentis? Is Sentis just a rebrand of Barracuda? Barracuda is already an on-device AI model inference solution.

1 Like

My best guess is if using copyrighted works in a data set without explicit permission is deemed illegal, most AI tools will disappear (including Unity’s), which is something to think about before basing your game and workflow around those tools.

1 Like

Exciting times. I’ve been using Unity since it was Mac-only, and this development is one of the most exciting ones I have seen in a long time. I’m particularly thrilled that there are capabilities for using AI in the executables, and that there seems to be a commitment for webGL support. Like others asking above, I am very interested in hearing about which training sets have been used to train the LLMs, since I also believe this is crucial for legal reasons.
I have also applied for the beta, and hope to get in.