A.I. generated art for game development

[dr4]

This is something I have been discussing a lot lately, I really would like to know the community’s position on using A.I. to generate game assets and images (as A.I. is available for both 2D and 3D), based on my discussions I can see that currently there is 2 main positions:

• We should use it because like any other technology it is a tool that allow us to get things done better, cheaper and faster.

• We should not use it because it essentially copies other artists arts, it take little effort and you should have no rights on whatever it creates.

As a software developer who has 10 years of experience writing code and 0 years on doing anything artistic A.I. art really got me excited, I could finally focus on my strength and generate any art I need, but seeing the controversy it feels like it could backfire, not only based on people’s opinions but on any copyright law that different governments create around that.

[neginfinity]

@dr4

• We should not use it because it essentially copies other artists arts, it take little effort and you should have no rights on whatever it creates.

I have impression that this position is held by people who do not really understand how image generators work and have no experience using them.

@dr4

As a software developer who has 10 years of experience writing code and 0 years on doing anything artistic A.I. art really got me excited, I could finally focus on my strength and generate any art I need, but

It is not this easy. As they say, there’s no free lunch.

It takes time to learn how to use Stable Diffusion, for example, and how to coax neural network into producing something similar to what you would want. The tools are not simple. What’s more, the tools are very rapidly evolving, for example, a short while ago someone introduced control net and now it is possible to pose characters. Well, sort of. The instrument can refuse to cooperate.

Basically, to get good results, you need to invest your time and learn. If you do not invest your time and learn and just slap a basic midjourney artwork, you’ll produce “fodder art” which will look cheap to people familiar with Midjourney/Stable Diffusion, despite being detailed. You know how some people used Daz3D to make comics and visual novel characters? It is this sort of effect.

Also, be aware that learning how to draw is an option. Your software development skills should aid you while researching a new field. Any new field, art included.

[DimitriX89]

The general consensus on gamedev forums is “use it and to hell with what the artists think; automation blah blah”, so to get the alternate perspective you’ll have to ask in multiple places

Personally, I am mostly negative to it, partly I do not have the hardware to get decent results from standalone version of Stable Diffusion, and refuse to support online services, like Midjourney, with money. It doesn’t exactly “copy” artists’ work, but it had artwork (scrapped from all over Internet) compressed into itself during creation. Since artists had no idea someone will ever use their work to train neural networks, no one could opt out of this practice until it was too late. So no surprise many of them beel betrayed. (AI advocates will go on about how “it Is tOTallyY LIke hUmANs learN to DraW”, no I can’t agree. Neural network doesnt form understanding of, say, anatomy and perspective from examples, and only draws what is statistically more possible. Thats why it is, for example, exellent with faces at front angle, but messes up eyes most of the time, because look direction on source pictures was all different)

About how it can help you, well, as neginfinity said, will depend on the context of what you need. It is trivial to produce something that looks like artwork, but for more specific tasks, you’ll have to fight against the algorithm. Issues with eyes, hands, poses and even elementary logic (it can draw the belt overlapping an arm or more than 4 legs on a horse). At some points, even hacks like in-paint or img2img may be insufficient, so manual retouching will be needed. Hard to say where it is going, both legally and reception wise. In the end I feel like consumers will decide, not nesessary creatives themselves

[AcidArrow]

@neginfinity

I have impression that this position is held by people who do not really understand how image generators work and have no experience using them.

I mean, there is a big discussion to be had if using copyrighted works in the training dataset is legal / ethical, especially in cases where you can make sure the works of some artist are over-represented in your output by adding “in the style of x artist”.

But yeah, I think some kinds of artworks that are easily reproduced by AI-Gen will soon be considered cheap. They will become the same as “helicopter shots” for movies. It used to be the mark of a big production to have helicopter shots (like following a car from high above), but now with drones, every no-budget third rate movie has a shot like that, and people are starting to associate those shots with low budgets.

[neginfinity]

@AcidArrow

I mean, there is a big discussion to be had if using copyrighted works in the training dataset is legal / ethical, especially in cases where you can make sure the works of some artist are over-represented in your output by adding “in the style of x artist”.

There’s not much to discuss here, as the training process is similar to what human does while looking at a picture. If you ban robots from looking at someone’s works and claim that this is copying, you should ban humans from looking at it as well. Because by this logic humans also “copy” the picture into their mind in the same way.

Styles are not copyrightable. And humans are perfectly capable of doing the same thing.

Another thing is, a single style is useless, people use blends of many styles that ends up resembling none of the original. So “style of X” does not even refer to the painter himself/herself. It is a bookmark with desirable behavior traits of neural network and it is meant to be combined with other bookmarks.

And that is the reason why I say “stealing artists works” is an opinion only held by people who haven’t seriously used the tool. Those people think their style has value, and everybody is out to steal it. It is similar to novice programmers thinking that everybody is going to steal their “extremely valuable” code. In reality their style has no value alone and is used as a keyword to make the neural net draw faces better. And when it comes to proper painting, a real artist worth his salt will run circles around stable diffusion.

Also, in case of stable diffusion specifically, rather than using style merging voodoo it is easier to load a different checkpoint instead.

Effect of checkpoint switching. The prompt is minimal.

[DimitriX89]

@neginfinity

There’s not much to discuss here, as the training process is similar to what human does while looking at a picture.

Neural networks compute difference between noise and a picture and store it, then remix those deltas. Nothing like breaking down the shape of object into simpler forms and forming understanding of general principles, like humans do (there is an element of it in drawing process, but it is a really small part). Humans do inevitably develop their own style, even they start from copying. AI have to utilize the style of existing artist for the best results. To say that the training process of neural network is similar is confirmation bias, and in your case, if you claim to “know how they work”, intentional misleading.

@neginfinity

If you ban robots from looking at someone’s works and claim that this is copying, you should ban humans from looking at it as well.

Woah, slow down there with false moral equivalencies. Robots are tools and do not have any rights compared to humans (otherwise, there is no sense of using the machines at all, as it would be equivalent of slavery). We should, no absolutely NEED to restrict what machines can and cannot do (because, if you ask me, the technology in general becomes more and more a trainwreck with the passage of time).

@neginfinity

Styles are not copyrightable. And humans are perfectly capable of doing the same thing.

There is a concept of unlawful use of someone’s likeness
https://www.dmlp.org/legal-guide/using-name-or-likeness-another
Prompt engineers leech from the popularity and relevance of existing artists all the time. Check out the story of Greg Rutkowski. Maybe flooding the internet with cheap knockoffs of his work isnt exactly “illegal”, but only because of it being gray area currently. (Also, pics “in the style of Rudkowsky” are good illustation of how “good” AI with anatomy and character interactions). Again, most humans who ever copied to study moved on with their art eventually

[Antypodish]

People been reproducing existing arts and their variations forever. In most cases there is no legal action against. Unless someone makes something like repaint of Mario animation nearly 1:1, with changed colors.

Art stations and other image stocks are flooded with iterations of other works. Appears there is no issue with it since years pasts. Before any AI even been to mentioned.

AI is not simple copy paste. It is unable to recreate exact image when asked directly. Unless specifically supported with such image of Monalisa for example, just to make her look cry.

Other than that, is just a tool and tech to adapt too. Learn it to use it. It isn’t tool that will make what you want with two clicks. As mentioned already, it requires learning process, to create right prompt, to use right models and their variations. It is tool that still requires engineering level and artistic expertise.

Or be like Codak, whoo decided not to go with progress of digital cameras, when times arived and basically it is left over in some narrow niche of photo making market, almost unrecognisable anymore.

Law as usually will be behind the tech. By the time it’s get to some consensus, tech will advance and law will be obscure. Most likely dead.

[AdrielCodeops]

@DimitriX89

Neural networks compute difference between noise and a picture and store it, then remix those deltas. Nothing like breaking down the shape of object into simpler forms and forming understanding of general principles, like humans do (there is an element of it in drawing process, but it is a really small part). Humans do inevitably develop their own style, even they start from copying. AI have to utilize the style of existing artist for the best results. To say that the training process of neural network is similar is confirmation bias, and in your case, if you claim to “know how they work”, intentional misleading.

Woah, slow down there with false moral equivalencies. Robots are tools and do not have any rights compared to humans (otherwise, there is no sense of using the machines at all, as it would be equivalent of slavery). We should, no absolutely NEED to restrict what machines can and cannot do (because, if you ask me, the technology in general becomes more and more a trainwreck with the passage of time).

There is a concept of unlawful use of someone’s likeness
https://www.dmlp.org/legal-guide/using-name-or-likeness-another
Prompt engineers leech from the popularity and relevance of existing artists all the time. Check out the story of Greg Rutkowski. Maybe flooding the internet with cheap knockoffs of his work isnt exactly “illegal”, but only because of it being gray area currently. (Also, pics “in the style of Rudkowsky” are good illustation of how “good” AI with anatomy and character interactions). Again, most humans who ever copied to study moved on with their art eventually

You could put a camera on a robot and train the model based on what it sees and you would have something very close to a human. The biggest difference is how humans, through their life experience (biological needs, cultural needs…), disseminate what is art and what is not. So supervised training AIs will have a human bias about “what is pretty to look at”. But it has nothing to do with tracing other drawings, in fact contemplating, copying and deforming is the basis of human creativity.

Try putting a newborn human being in an empty white room, without even noise propagation, let it grow there for 18 years. Then give him a pencil and a sheet of paper. Let’s then see how it “generates its own style”. To think that humans would be able to create from nothing is absurd. One would have to assume some kind of mystical soul with no scientific basis.

[AcidArrow]

@neginfinity

There’s not much to discuss here, as the training process is similar to what human does while looking at a picture.

You’re right, if you’re making such broad equivalences and presenting them as absolute truth, we don’t have much to discuss.

[neginfinity]

@DimitriX89

Nothing like breaking down the shape of object into simpler forms and forming understanding of general principles, like humans do

Nope.

Humans do not do that sort of thing while looking around.

You see an image once, you have it in your brain. How well you can recall it depends on your visual memory, for example, in my case the image will be “blurry”. But it is just there. Without any in depth analysis. You saw it, you copied it. It is encoded somewhere within your circuitry. If you’re good enough, you can copy it back to paper. People with photographic memory exists, and there’s opinion that images are actually stored in photorealistic quality, it is just you can’t access them easily. Sure, there’s a tiny number of people without ability to think in images (called “aphantasia”), but for majority of people you can evoke an image in your mind and play movies there.

@DimitriX89

breaking down the shape of object into simpler forms and forming understanding of general principles, like humans do

Again, nope.

Because stable diffusion is tiny, it IS breaking down the shape of object into simpler forms and forming general principles. Just like you described.

Stable diffusion forms a library of patterns, and it is blending them based on input to produce final image. To notice that, you have to play the prompt while keeping the same seed and see several thousands of images. A casual user will never get there.

Basically, if you generate many pictures with the same seed, you’ll notice that, for example, on the first image there’s a crevice on clothes. In the second image the crevice forms into someone’s neckline. Then it becomes a branch. Then a river. Its shape remains the same and position remains the same. Sometimes it is barely visible. Patterns like brick wall fade in or fade out of image based on prompt produced, but while fading in they do not move from where they are. There’s a lot of those patterns, and the image is formed by overlaying them. You see this in greater detail when someone tries to use stable diffusion to animate a video, because patterns noisily move between frames and become more noticeabble.

And that is generalization in practice. Because it is impossible to store terabytes of data within 2 gigabyte file (minimum size of stabble diffusion brain), the net has to generalize. It wouldn’t work otherwise.

@DimitriX89

Woah, slow down there with false moral equivalencies. Robots are tools and do not have any rights compared to humans

Your visual cortex is a tool and has no rights compared to a full human. The process is identical therefore restrictions should be identical. Anything a human with a neural network can do, an artist is capable of doing and can do better. If robots are not allowed to generalize based on your works, then humans shouldn’t be allowed to do it either, because the process is the same, and the end user of the data in both cases is a human.

@DimitriX89

if you ask me, the technology in general becomes more and more a trainwreck with the passage of time

Mankind is a trainwreck, not technology. On ChatGPT subreddit someone seriously tried to argue that GPT-3 has feeling, will remember everything told to it and therefore we should be polite to it so it won’t kill us all in the future. That person was completely serious. That is a high degree of technological incompetence in practice.

People saying that “neural net is copying” are the same as that guy. In essence, those sort of groups will keep popping up and in practice they seek to stop technological advancement. If those guys gain power and succeed, the world will become a place I’d rather not live in.

@DimitriX89

There is a concept of unlawful use of someone’s likeness

Someone’s likeness is their face and is not their drawing style. And you can’t copyright your face anyway, because it was not created.

@DimitriX89

Prompt engineers leech from the popularity and relevance of existing artists

Before I tried stable diffusion I did not know that Greg Rutkowski existed. Like many other artists. Art, in the first place is a niche interest, and most people do not care about it.

What is happening is that those artists become known through prompt engineering. It is advertising.

@DimitriX89

Maybe flooding the internet with cheap knockoffs of his work isnt exactly “illegal”, but

Dude. People are not trying to make his knockoff, because in the first place it was never about rutkowski.

Making a “knockoff” will be worthless, and rutkowski can run circles around any neural network output, which is something you’d known if you checked his real artwork.

The idea is to have a convenient keywords that improves image quality. A convenient keyword in this case happens to be artists name. It doesn’t matter what keywords mean, and whose name it is. What matters is its effect. It is a shortcut.

[DimitriX89]

@AdrielCodeops

Try putting a newborn human being in an empty white room, without even noise propagation, let it grow there for 18 years. Then give him a pencil and a sheet of paper. Let’s then see how it “generates its own style”. To think that humans would be able to create from nothing is absurd. One would have to assume some kind of mystical soul with no scientific basis.

You are putting words in my mouth. Where did I said about “creating from nothing”? I meant that the human creative process is different from the neural network, and that the argument that they are the same as a defence of AI art is gaslighting and wishful thinking. Neural network model is unchanging after it is created, while humans learn during all their lives. It can never achieve understanding of factors such as shape language, physics and anatomy to be able to move away from simply reusing styles. Thats why I say it is more of equivalent of compression.

And about “believing in mystical soul”, well, funny, because the AI community has many trait of a religious cult by itself. Good portion of the AI proponent’s arguments is ascribing personhood and agency to a machine, and also believing humans/biological life be imperfect and in need or replacement. In other worlds, attempt to produce a god to rule us all, which is the root of the movement for general AI development. Not any real world concerns, just the idea that AI will know all, invent everything, solve every problem, etc, if you give enough money to tech companies. This is no more than faith. Assuming that there is a soul is as biased as assuming that human is equivalent of machine. Since in the end, our scientific understanding of reality is but a model, which sometimes works, but nothing close to completeness or absolute precision

[neginfinity]

@AcidArrow

You’re right, if you’re making such broad

I’m stating my opinion, which is a result of my life experiences and values.

If someone is unhappy about my opinion and beliefs, well, tough luck. You’ll have to learn to live with me existing.

Also, writing this all down takes time. If, after me spending time to carefully break it all down, and explain piece by piece why I think this way, someone still isn’t getting it, well, at this point this is no longer my problem.

Sufficient effort has been made, if the message isn’t getting across, I can’t help you. If after detailed explanation all I get is an attempt at sarcastic ad hominem in return, then a dialogue was never an option in the first place.

[dr4]

@neginfinity

It is similar to novice programmers thinking that everybody is going to steal their “extremely valuable” code. In reality their style has no value alone

This is similar to a comparison that I have been doing, in the past decade tools to create things without any need of code exploded, Unity itself tries to make that happen, but no one really cared about developers (mostly because they were developing those tools to start with), I understand that it is hardly a fair comparison, but often you can write a whole piece of software with no code to be exactly what you want, but you can’t create a whole comic book to be exactly what you want by using any kind of tool, even with A.I. more often than not you will need to use what you are given instead, so artists with that power will not go anywhere in the short term, I’m aware that many of them are using A.I. generated art to create a base to work with and adapt it to their needs, speeding up the process a lot.

[neginfinity]

@DimitriX89

I meant that the human creative process is different from the neural network, and that the argument that they are the same as a defence of AI art is gaslighting and wishful thinking.

You’re thinking in the wrong direction.

There’s no creative process in neural network, because it is incapable of doing anything by itself. The one performing the creative process is its operator. The network can produce a highly unexpected but interesting result, which can steer further process in a new direction, but that’s the extent of it.

Neural network only knows how to draw. And the way it learns to draw is similar to the way humans learn to draw.

It is not a human. It is not a human artist. It is an external art knowledge module for your brain. If you’re an artist, you’ve developed an organic equivalent in your brain. If you’re not, then you can use the network.

@DimitriX89

Neural network model is unchanging after it is created,

It is not unchanging. Look up Textual inversion, Fine Tuning, and LoRA. You can alter it. You have to train it. Because it has no eyes and cannot do it by itself. It has no agency, remember? So if you want it to learn something new, you have to help it.

@DimitriX89

Good portion of the AI proponent’s arguments is ascribing personhood and agency to a machine, and also believing humans/biological life be imperfect and in need or replacement. In other worlds, attempt to produce a god to rule us all, which is the root of the movement for general AI development.

You’re making a lot of accusations here, without supporting them with anything. Saying that the root of movement for making an AI is trying to produce a god is rich.

And in the end you yourself speak from position of faith.

[andyz]

I expect this thread to be shut down as not very Unity specific and usual arguments ensuing, but…

It remains to be seen how the legal situation pans out - perhaps there is just not enough similarity, in images generated, for any of the suits in progress to be successful at the end of the day (see https://stablediffusionlitigation.com/ Getty Images etc.)

If you use these tools you can expect to find them error prone because whatever is said they seem unable to learn basic anatomy! Artists will also be quite unhappy if you use them - it is understandably a threat to their work

[DimitriX89]

@neginfinity

Humans do not do that sort of thing while looking around.

ARTISTS do that while creating artwork. Important distinction. While you can say imagination is roughly similar to what neural network does, it is really a small part of drawing and painting process

@neginfinity

Because stable diffusion is tiny, it IS breaking down the shape of object into simpler forms and forming general principles. Just like you described.

If it was true, then Stable Diffusion would have no problem with drawing hands, eyes, four legged animals or character interactions whatsoever. If minute movement of the eye in the training data (as shown in your last picture) can confuse it, what general undestanding is there to speak of? Efficient compression algorithm is still compression

@neginfinity

Your visual cortex is a tool and has no rights compared to a full human. The process is identical therefore restrictions should be identical. Anything a human with a neural network can do, an artist is capable of doing and can do better. If robots are not allowed to generalize based on your works, then humans shouldn’t be allowed to do it either, because the process is the same, and the end user of the data in both cases is a human.

My visual cortex wont work without other systems of the brain, and brain is required to be full human. While it is perfectly possible to live without using artifical neural networks, even if it is harder. (Again, you are passing an assumtion that those processes are identical as absolute truth). Idea that machines should have same rights as person is idealistic to absurd degree. Then what, we should stop selling and owning them? Bye the capitalist economy, and the AI development along with it. Or maybe we should have car drivers compete with Olympic runners, because a human is in control in both cases.

@neginfinity

Someone’s likeness is their face and is not their drawing style

It does extent to using their name.

@neginfinity

Mankind is a trainwreck, not technology. On ChatGPT subreddit someone seriously tried to argue that GPT-3 has feeling, will remember everything told to it and therefore we should be polite to it so it won’t kill us all in the future. That person was completely serious. That is a high degree of technological incompetence in practice.

Sorry, but you arent helping this case. Saying that we should treat humans and machines equally in the very same post, and also implying that the technology is perfect if it wasnt for those pesky humans. Machines arent perfect, they are merely more specialized. Such religious mentality is why AI can become a threat, in my opinion. Putting priority on the tool before its user will guarantee general AI, if it is possible, wont work in our interests

@neginfinity

Before I tried stable diffusion I did not know that Greg Rutkowski existed

People wouldnt start generating such art if Rutkowski wasnt relevant in the first place

[DimitriX89]

@neginfinity

You’re making a lot of accusations here, without supporting them with anything. Saying that the root of movement for making an AI is trying to produce a god is rich.

“We need an AI governer. Which will be immortal and capable of increase its intelectual capability infinitely”
Sorry for misquoting it, but the gist is there. And for some it is even not a joke

[MadeFromPolygons1]

Regardless of the discourse here around legality, morality etc - you cannot stop progress.

Either get on the train, or get left behind. You can either be one of the artists who adapts and incorporates this into their workflow, or become one of the artists considered to be doing “retro” style workflows in the future.

If anybody here genuinely thinks that legal or moral objections will somehow stop this technology being used globally and in every industry, then you are not paying enough attention to what is happening in the software industry right now. Every agency and company I work with currently, has pivoted to incorporating AI to some degree.

It sucks, we have artists in my company and we dont like it, but its the reality.

So for those that hate it - Time to start learning how to use it instead of arguing against it.

[neginfinity]

@DimitriX89

If it was true, then Stable Diffusion would have no problem with drawing hands, eyes, four legged animals or character interactions whatsoever. If minute movement of the eye in the training data (as shown in your last picture) can confuse it

You aren’t seeing the forest for the trees here.

“would have no problem” is non-sequitur. Process is the same, memory capacity is inferior to a human artist, hence the artifacts. It still manages to generalize and be usable. It is usable because it generalizes. If it didn’t generalize it wouldn’t be able to draw at all. It has thinking capabilities roughly equivalent to a honey bee, after all. The model is TINY. The fact that it can process so many image is already miraculous, and there’s no way to compress data into it. The training data is 100 terabytes, and 2.3 billion images.

If you want better quality, well, you need more memory, hardware power, and bigger training database. Midjourney offers that.

@DimitriX89

Idea that machines should have same rights as person is idealistic to absurd degree.

That’s a strawman. Or you aren’t listening. What rights? Stable Diffusion is not a person. And neither is your visual cortex.

However, if you’re not allowed to use an image for training stable diffusion, you should not be allowed to use it with your visual cortex for training your mind. Meaning you should not be allowed to see any copyrighted artwork, because in your case fine-tuning of your circuitry is going on all the time non stop, meaning you’re unlawfully copying every single image you see.

The artists are trying to implement absurd restriction. I’m demonstrating absurdity. It is worth noting that copyright laws have been steadily increasing duration of copyright protection in the past 2 hundred years, and this is the point where they’re taking it too far. In the end the only thing it is gonna achieve is that another country will take the lead in neural network research.

@DimitriX89

It does extent to using their name.

You can’t copyright names either. Look it up.

@DimitriX89

Saying that we should treat humans and machines equally in the very same post, and

I have never said that, though. You’re making a strawman. Either deliberately, or you need to work on reading comprehension.

[DimitriX89]

@MadeFromPolygons_1

Regardless of the discourse here around legality, morality etc - you cannot stop progress.

Either get on the train, or get left behind. You can either be one of the artists who adapts and incorporates this into their workflow, or become one of the artists considered to be doing “retro” style workflows in the future.

If anybody here genuinely thinks that legal or moral objections will somehow stop this technology being used globally and in every industry, then you are not paying enough attention to what is happening in the software industry right now. Every agency and company I work with currently, has pivoted to incorporating AI to some degree.

It sucks, we have artists and we dont like it, but its the reality.

So for those that hate it - Time to start learning how to use it instead of arguing against it.

The technological progress isnt an objective good, it is increasingly a liability (considering the waste of resources on unnesessary or even harmful projects). Sorry, but “you cannot stop progress” is a cargo cult mentality to be honest; we need to evaluate all the technologies on case by case basis. Many seemingly promising technologies fell into obscurity, so it can go either way. In the end, I believe the consumers will decide, but the pushback is justified

One of the reason that it image generation is hated by artists is that in many cases it is hard to incorporate it into existing content creation pipelines. Since it is quite rigid and limited. Either you settle with the first draft and close your eyes to errors, or lose the speed benefit while trying to fight against an algorithm.