Is it possible? Can you give me an insight?
Thank you for the question! At this moment, not possible yet.
What do you have in mind? Or what is it that you are trying to create?
turnaround 360 degrees spritesheet
I have to add that the ultimate goal is to input the images to NERF and produce a 3d model.
I see! Can I also clarify if there’s a specific class of sprites that you are looking at? E.g. Characters (bipeds, quadrupeds), environment, props etc.
Fictional characters.
The rest props I scan them from real objects with my smartphone and use NERF.
If you want my opinion, Unity CAN be a pioneer in fictionTO3d.
While now it is possible to do realityTO3d with NerFs or Gaussion Splatting, I havent found yet a solution to rotate 2d sprite or SFW or NSFW (yes there are people that do Adult games like me) high quality fictional characters in a 360 manner. I tried A LOT like stable-diffusion A1111 and many others, with different as good as I can prompts, and I cannot yet manage a 360 or 720 degrees turnarounds, mainly cause I can’t achieve consistency in characteristics of the fiction character.
As I can see Muse sprite has a lot of potential. Imagine a 3D fictional character sprite for example. That would evolutionary! It will skyrocket the use of Unity AI when this thing goes live.
Also increase the Muse training photos to be trained for a custom style from 10 to a lot more.
Thank you for the additional details! Yes, I can imagine how character sheets can be very useful for many use cases. Let me add this feature request and use case into our ProductBoard and share them with the team.
…mainly cause I can’t achieve consistency in characteristics of the fiction character.
Just to clarify, do you mean that the character looks different in different angles e.g. frontal, profile 3/4, 7/8 etc. or even in different poses?
Also for custom style training, the reason why we set a cap at 15 is that the optimal number of training images is between 5 to 15 images, above that the training time increases a lot for minimal improvement.
Curious also, how many images do you typically have in a training set for a single character?
Yes the character looks different in all angles:
- The consistency can be achieved through extensions in Automatic1111 and ContolNet “reference” pre-processing. But if you move even one angle with OpenPose extension (biped) it produces altered results in clothes for example. This leads NerFs or Gaussian splatting to not be able to recognise the fact that it is the same character and produce wrong Point Cloud in a photogrammetry way.
- The prompt CANNOT be consistent. For example: If I prompt a necklace for the character in front view, when the biped batch reference reaches 3/4 or even back view, the necklace will be present at the back or side view exactly as the front view of the character. And that happens FOR ALL the characteristcs such as bra, clothes, belt, body structure. Condescending you must produce 360 images with camera low, mid, above the character (total 1080 images) by breaking the spritesheet to chunks, there is absolutely no way to batch handle the prompt.
500 1000 steps for a new style in Automatic1111. I have 4090 24GB and a good PSU, UPS, liquid freezer cooler, so I can handle them. What I didn’t inspect is if the generation and the training is proccessed by my machine or a bigger graphics card like A100 eco-system online hosted by Unity. I have to check it.
Whoa 1080 images! That sounds very intense. I can imagine how scalable workflows can be very helpful in such cases.
Well, they are not that many. My rtx 4090 can batch generate them (1024pxx1024px, 150 steps) in about 20 hours. I have generated about 2M images the last two months.
- Batch generation is achievable.
- Loopback script is achivable, so you can pre-proccess reference with ControlNet the last generated image.
But it can’t be prompt consistent.
If you manage a consistent character turnaround spritesheet generation, you can create literally ANYTHING with Muse Sprite. That feature doesn’t exist yet, in history of AI.