Aify - Text to Image Utility [Editor Extension]

Community Showcases
, ,
1

Unity’s first Editors Extension asset containing neural network models and libraries that are capable of image generation through text (Image to Text), or images to enhanced images using AI (Image to Image) and deep learning.

You can create an unlimited number of images/textures in Edit Mode or Play Mode (runtime) inside the Unity Editor. This asset has a one-click, beginner friendly GUI and does not require any coding to use.

Please note: All the artwork you see on this forum post is AI generated.

The AI models that come with this asset are:

  • SRGANs
  1. Light Weight Super Resolution Generative Adversarial Network - Local Python Inference
  2. Heavy Weight Super Resolution Generative Adversarial Network - Local Python Inference
  • Depth Estimation Model for AI Depth and Displacement Maps - Server Inference
  • Text to Image Diffusion Model - Server Inference

Dependencies:

This tool requires the Barracuda package from the package manager and an active internet connection for some features.

Additional Features and Applications:

️ Text to Image Converter: The desired image/texture should be described, and the results can be obtained within seconds through a single click, utilizing fast and efficient inference.

Examples for concept design:

● Octane rendered character portrait of mitsurugi, 3D, octane render, depth of field, concept art, vibrant colors, glow, trending on artstation, ultra high detail, ultra realistic, cinematic lighting, focused, 8k

512×512 290 KB

● Interior design, open plan, kitchen and living room, modular furniture with cotton textiles, wooden floor, high ceiling, large steel windows viewing a city

512×512 61.7 KB

● A highly detailed epic cinematic concept art an alien pyramid landscape , art station, landscape, concept art, illustration, highly detailed artwork cinematic, hyper realistic painting

512×512 80 KB

Examples for Texture Design:

9 Seamless AI Generated Copy Right Free Textures with their corresponding generated normal maps

Rough Sketches into textures or icons: Convert a sketch into stunning and stylized game art using the Match Image Feedback Loop which is an image to enhanced image converter.

907×539 160 KB

Creation of Materials: You can use the normal map and specular/smoothness map generators to add additional details to your material without an external 3D software.

738×712 577 KB

Generate Game Icons: Generate unlimited 3D game icons for menu inventories, menu options, avatar icons, UI buttons and more!

1024×512 483 KB

Create Backgrounds and Characters for Visual Novel Games: This asset can create visual novel artwork without you hiring a 3D artist. Just put in a text prompt and let Aify take it over to generate art.

1950×1300 761 KB

Upscale Game Textures: The asset offers the ability to upscale low-resolution textures to 4K textures, providing users with enhanced texture visuals. Using the barracuda inference python library, this tool can improve the overall quality of textures by bringing out details that may have been lost in low-resolution versions. With the assets efficient and user-friendly interface, users can quickly and easily upgrade their visuals with high-definition textures in a click.

1920×800 223 KB

Editor Script: The Editor Script displays all the options neatly in one panel. There is a batch mode as well as a list of all the settings you can choose from in the editor. The editor also has a list view of all the maps you generate with the post processing settings automatically in a single click.

1024×512 111 KB

Keeping it all in the editor: Keeping all assets in one workspace inside the Editor and having to switch to fewer services can have several benefits, such as:

  • Improved Efficiency: When all assets are located in one workspace, it becomes easier to access and manage them. Users do not have to spend time switching between different services or applications, which can be time-consuming and lead to a loss of productivity.

  • Streamlined Workflow: Having all assets in one workspace can help create a more streamlined workflow. This is because users can easily move between different assets, such as code files, images, and documents, without having to navigate between different services. This can help to speed up the development process and make it more efficient.

  • Reduced Complexity: Using fewer services can help to reduce the complexity of the development process.

Please check out the documentation for an in-depth explanation and working of the asset.

4 Likes
2

Reading the asset store page: No sign up, No API Keys, no recurring payments, no subscription fees, no additional costs, no restrictions, just one-click easy to use unlimited inferences on our model.

It looks like a great set of workflow tools.
I’m curious, how are you able to provide this for free? (After initial purchase that is)
I have to imagine the upkeep, electricity, and wear & tear costs on the hardware wouldn’t make this profitable. But you’d know better than me in this case - but it’s definitely unconventional to offer a service like this for a one off fee.

8 Likes
3

Wondering the same question! Is this running locally on my device? Can it support mobile and mac?
I’ll buy this plugin in no time if there are more details about it!

1 Like
4

I bought the module. Currently, I am getting a lot of errors and I have tested it on 2 different computers and in 3 different projects. The image match simply does not work. I get an error every time. Same thing for generating a dept map. An error each time (my internet works perfectly).

I could not make a Rough Sketches into textures or icons, because it is not possible to make an image to image. You have to enter text. Unity crashed several times too because of memory problems (my computer is very powerful so it’s weird).

I was not able to upscale like this image for example :

1200×800 164 KB

Maybe a more complete tutorial to show how to do what it’s supposed to do :
Create inventories
Generate 2d Assets
Generate materials
Feedback loop generation
Create artwork
Upres textures

For the moment, I can’t do what I want with the tool without errors. I will wait for updates and examples.

3 Likes
5

Hello,

Thank you for trying out this package.

Please could you elaborate on what type of errors you are getting when performing Image to Image conversion. Please also inform us about the version of Unity and Barracuda you are using. The errors could be due to a version mismatch. One of the developers encountered errors with generating an image with the text prompt. The issue was that in the setting “Allow downloads over HTTP” setting was disabled. We assume there could be an issue in uploading the image as well. Please try toggling this setting and try again. We are discovering errors like this and trying to rectify them.

You could also go through the troubleshooting section in the documentation. That may be helpful in identification of the problem (for example if the uploaded image was NSFW).

As of now, we know that Unity crashes if the images of dimension 1024x1024 (with round dimensions enabled) or more are tried to upscaled by 4x making them 4096x4096 at Light Weight Super Resolution (LWSR). Please tell us if this was the case for you.
We are also currently building a server based SRGAN upscale model with much better results rather than a locally run model.

Again, thank you for taking out the time to explain to us the issues so we can fix them for you in newer iterations of the product.

6

Hello Fyrburn,

Yes, indeed the asset does not have a sign up, or API keys on any other type of payments.
Our server is based on a Amazon EC2 P3 with Tesla V100 GPUs instances that provide high compute capabilities at a reasonable cost. We will be upgrading or making changes to the server settings regularly to ensure fast inferences on our model. Paired with this, we have implemented spot instances for improving cold start performance and reduce costs.

Our inspiration for this project was the thread where AI art for games is discussed in detail. We believe that this could be a very useful tool in generation of seamless textures, 2D game assets, UIs, concept art and generation of new ideas and a huge help for creators. We also offer upscaling of images inspired by the thread - Can indies use this AI texture upscaling to boost their game content? which would help developers use upscaled AI generated textures and boost the art designs. This has impressive implications which we would like developers to use with ease all inside Unity.

7

Thank you for your reply.

Unity : 2021.3.22f1
Barracuda : 3.0.0

No matter which image is used in “match image”, I always get the error There was an error in generating the image. If I remove the image, it works (not always though). Also, I was wondering what do I have to write to this location to be able to do Feedback loop generation?

No matter what image I use too, I always get the error “There was an error in generating the dept map” when I try Generate Dept Map.

No matter what image I use as well, I always get the error “A Native Collection has not been disposed” when I try “Upscale Resolution Settings”.

I can’t do anything that is advertised except generate an image from a text. However, since my goal is to modify existing textures, I can’t do anything at the moment unfortunately.

I don’t find the option Allow downloads over HTTP in Unity 2021. I seems it was added in 2022.

1 Like
8

I don’t know if it’s based on stable diffusion. but if yes, can you add a model for example from civitai.com
and we can choose a model from the editor, so that the resulting image can be better.

9

Yes, of course, that is on our agenda. We will be improving our model so that it can generate better results.

1 Like
10

Thank you for taking the time to upload the screenshots and outline the issue.
We tried to replicate the errors, but were not able to unfortunately.

We used:
Unity: 2021.3.5f1
Barracuda: 3.0.0

The inferences for Match Image worked perfectly.
We also tried using transparent PNG images. The image generation filled out the transparent background, but the connection to the server was established and the response was received. Same was the case for JPG/JPEG images. We tried images of resolutions 256x256 as well and images with import settings set to Sprite (2D and UI)

We had no issues generating depth maps either. However, there is still an issue with sRGB import settings which makes the depth map perceive the image as slightly darker as pointed out by another one of the developers. This will be fixed in the upcoming updates. However, again, we did not experience connection issues.

For upscaling, we receive a few warnings for unreferenced and undisposed data and occasionally the error of undisposed native collection, but the upscaled image is generated and saved.


The gif above is the intended functionality on Unity 2021.3 with Barracuda 3.0.0.

However, we will keep working on looking for a possible solution to the problem.
Could you please try a non-transparent image of resolution 512x512 with default settings and tell us if that works with Match Image functionality?

11

Hello HelixNGC7293,
The upscaling models are developed with the help of pytorch which are converted to an Open Neural Network Exchange library (ONNX). Inferences on this model can be performed by the barracuda package manager dependency. These models are run locally. We provide two models namely the light weight super resolution (LWSR) and heavy weight super resolution (HWSR) models. The text to image models are run on a server due to their size and high computational requirements. Same is the case with depth maps.

As for mobile and mac support, we have not tested inferences on those platforms. We do not suspect there would be any issues with accessing the text to image model or the depth map generation model though as they utilize an external server.

1 Like
12

Hi there, I bought this yesterday too - the idea of this is something I’d love to support! What would help a lot is some step by step examples of how to achieve each of the things in the docs. There are pictures which are about the same as those on the asset page, but the ‘how’ is missing for me. This is how I tried your ‘sheep’ example:

  • I took a screenshot of your ‘sheep’ picture. The ‘generate image’ button remains ghosted. Is this intended?
  • I type ‘Sheep, game art, drawing, cartoon style’ in the prompt box, now I can generate. It generates a slightly improved sheep under ‘Selected Textures’.
  • I click ‘last’ next to the match image, and it pulls the last generated sheep up for me to repeat the process. It wasn’t really converging to an improved sheep though, but that may be my bad prompting.

Questions:

  • Am I doing it right?
  • Is it correct that I get only one image from the matching?
  • I can get multiple images from ‘browse concept art’ - is it possible to then mutate those images? I can’t seem to copy them, view them larger or move them into the ‘match image’ box

A good example of easy to follow ‘recipes’ is this one: Recipes | TopDown Engine Documentation. I think a recipe for each key function would be really helpful to get us underway.

Thanks!

13

Thank you very much for describing in detail the problem and the expected outcome. It is clear to us what you require help with and what you would like out of this asset. To quickly answer the questions you asked:

  1. “I took a screenshot of your ‘sheep’ picture. The ‘generate image’ button remains ghosted. Is this intended?”

Yes, this is intended. Without a prompt, the model does not have a direction to take to enhance the image. For example, you want the sheep to have a particular art style, cartoonish, photorealistic, rendered, drawing style or for it to look fluffier, skinned, have tinted wool, and other photo details like have a background, shadow, and shaded.

  1. “I type ‘Sheep, game art, drawing, cartoon style’ in the prompt box, now I can generate. It generates a slightly improved sheep under ‘Selected Textures’.”

A bit more detail could go into the prompt, for example, in the sheep example, we had something along the lines of “3D render of a sheep with big horns, red eyes, muscular goat, bison, octane render, plain white background, shadow, minimal, 8k, raytraced, vray”. To get the image in this forum post, we needed to perform 6 iterations of the process. Some others can take up to 10 or more iterations with branching. If you want more variation in the image, you can decrease the Match Image Strength parameter and the CFG scale. If you want only a slight modification, you can increase the aforementioned parameters.

Another point to add is that there is an option for you to preview a larger image under the Generate Image button. That way, you can see the generated image right inside the Editor.

  1. “Am I doing it right? - Is it correct that I get only one image from the matching?”

The prompting could be more specific and you could look into changing parameters to guide the image feedback, but yes, match image generates only one image at a time allowing you to “branch”. Branching here refers to guiding of the feedback loop by re-iterating if there is a wrong artefact that has a tendency to propagate over generations.

  1. “I can get multiple images from ‘browse concept art’ - is it possible to then mutate those images? I can’t seem to copy them, view them larger or move them into the ‘match image’ box”

Yes, apologies, this functionality is still under development. We added the Concept Art feature just recently. We will be supplementing it with a save feature in the next update of the asset.

If you are on Unity 2020.3, you could uncheck the fit grid option above browse concept art and increase number on the slider to get a close to maximum resolution image (512x512) from the editor and take a screenshot. But yes, it would certainly be easier to have an option to save the image.

As per your suggestion, we will also be implementing a "recipe for each key function so that it is easier to utilize the asset to the fullest capacity!

1 Like
14

Excellent Xpert3D, thanks for that thorough response!

  • Requiring a prompt makes lots of sense, thanks for sharing the one you used. I got a much nicer result with that.
  • It’d be great to be able to generate more than one image, say 3-5, then select each one to view it full size if needed. Then you’d have a button or context menu on each saying ‘Set Match Image to this one’ or whatever.

Viewing the image larger is very handy. That’s one interesting head my sheep has… :slight_smile:

8917721--1221548--upload_2023-3-31_22-35-14.png
8917721--1221548--upload_2023-3-31_22-35-14.png725×736 197 KB

1 Like
15

Hi, that’s a pretty cool asset, but I’m wondering, how will you guys be able to pay the server costs forever when the asset is single-time-pay model? What will you do if the sales go down or just stop for whatever reason? Can the people buying this asset be sure that they will be able to use this asset forever as is intended to be?

I still think that this is pretty nice, but I’d be a bit worried that is tied to your server so if you guys don’t succeed you could be shutting it down and we’ll be losing the money. Do you have a solution for that?

16

It is very strange, because I can generate an image with instructions, it works, but as soon as I try to use the same image for a “match image”, it does not work. I tried a PNG and JPG as well.

I tried it at work and at home to make sure it wasn’t a network restriction. It’s really strange!

17

Hello VenkoGames,

We understand your concerns. We developed a one-time pay model at a competitive price after achieving a breakthrough in increasing our cloud computing efficiency tremendously. Our inference speeds are unparalleled across any product on the game engine market as of the time of this post. This may change with time, however, we will keep updating the asset over time to make it more competitive. The models are huge in size, but since they are stored on our servers, this Unity package occupies only 40MB of disk space. Added to this, one major advantage of using such servers is that it does not need high end GPUs to run the model.

We realized that the only other asset that offer text-to-image functionality requires a sign up, API key integration and subscription fees. This is not only cumbersome to set up but also limited in usage.

In the unlikely event that our servers fail to operate in the future, we are developing models that can run on your system locally. In fact, we already have two trained AI upscaler models called LWSR and HWSR that work locally, without the need for servers or an internet connection to upscale images.

Hope this answers your question.

3 Likes
18

Do not forget that the size of image in pixels for “match image” option need to be in the multiples of 64.
For example 512x512, 1024x1024.
This asset is really breathtaking - gives excellent results.

1 Like
19

8918960--1221734--Orc01.jpg
8918960--1221734--Orc01.jpg2048×2048 650 KB
A quick example made with “match image” option and upscaled 4 times.

2 Likes
20

Thank you! We are glad that you liked the asset.
In the next update, that will hopefully be coming out next week, we will be releasing the “Match Image” feature which can take the height and width dimensions in multiples of 64 separately. For example, you will be able to generate images of dimensions 448x320 (landscape) , 512x256 (wide landscape), 320x384 (portrait) and so on. This can be further upscaled by a factor of 4 which will take them up to 1792x1280, 2048x1024, 1280x1536.