Stable Diffusion XL 1.0 to Unity

Unity Engine
,
1

I would like to have a version of Stable Diffusion XL 1.0 working in Unity. There is a Base (stabilityai/stable-diffusion-xl-base-1.0 · Hugging Face) and a Refiner (stabilityai/stable-diffusion-xl-refiner-1.0 · Hugging Face) part. I got the model working in Python and now my next step would be to get this working in Unity too. So I first was thinking I would just get a Base.ONNX and a Refiner.ONNX, but when I look at the HuggingFace websites I see that they have multiple ONNX files (text-encoder, vae, unet etc.) and now I am a little confused on which ONNX files I should use and how …

If anyone can give me a hint on how to approach this, I would really appreciate that. I am still a noob when it comes to AI (only played with tools, not coded against it yet).

Thanks in advance!

3 Likes
2

Im also trying to figure out how to implement and use Stable Diffusion with Sentis. Following this thread to see if anyone has any extra info and advice! :smiley:

2 Likes
3

Hey there - Thanks for identifying this issue. We plan to address this issue in 2 parts:

  1. Writing some sample or docs to assist in implementing Stable Diffusion “style” models
  2. Updating the API to support string conversions

We will let you know when we have an estimated date for this fix, known internally at Unity as issue 109.

6 Likes
4

Highly appreciated! Looking forward to that.
Will continue to improve my Machine Learning / Artificial Intelligence knowledge in the meanwhile.

1 Like