Artifical Intelligence Design of 3D games, Natural language prefab design

News & General Discussion
1

Today artificial intelligence is being used for books and concept art. Text to image generators are booming exponentially with projects like MidJourney, Imagen, Dalle2.

Soon 3D tools will become available which generate meshes including shaders, bump maps and texture mapping.

1896×938 617 KB

Polygons and 2D images have similar data volumes for machine-learning models, the difference is that TurboSquid, Pixar and other 3D mesh databases haven’t published their catalogues for study by AI. When folk leak 3D mesh libraries for AI, we will have AI engines that reduce many days of 3D mesh design to just couple of clicks.

Possible timeline of AI game-design tools:

2023: 2D animated sprite library is published with millions of 2D trees, creatures, buildings.
2024: A team downloads and leaks 10 million 3D meshes for machine learning
2026: First signs of mesh-with-shader generators, including bump maps and normal maps
2028: Animated mesh generators begin to specialize into AAA game tools: Trees, Creatures, Humanoids, Landscapes, Buildings.
2029: A team leaks a databank of 3000 2D platformer game videos, including the input data from game controllers.
2030: Game engines include AI animated-mesh-tools for select-and-place game design.
2033: Natural language 2D platformer design studio is published including sprite, movement, sound design
2035: The first AAA game designed by 2 developers hits the scene… Indie studios compete with major studios like Ubisoft, Blizzard.
2038: Laws are passed to stop children dying of exhaustion from the addiction of virtual-living.
2043: Worldwide epidemic of hyper-addictive games sweeps the world, many governments enact laws to try to cope with the epidemic.
2045: Hollywood becomes redundant. All movies can now be designed by indie game designers using physics based game-engines.

How fast will it really happen? What am I missing? could this really be the future?

Just fooling around, I think Unity3D of the future will let you select a space on your map and type “technological cat-woman with spider body” and “Amazon jungle with abandoned mothership, dramatic lighting”, and the asset store will become full of AI design plugins, ranging from 2D/3D weapons to aeroplanes.

I’d imagine that the game design tools and metholodolgy will always be specialized and additive, keeping editorial control of entities arranged manually using a scene editor system like Unity3d, using point and select interaction to populate move and edit AI generated prefabs, including manual scripting of logic like health-points and spawn-logic.

The future limiting factor is the amazing talent of coders required for that emerging field. They will be instrumental in creating open-source AI design platforms that load multiple AI text-to-prefab tools: Tree/Humanoid/Monster/Mechanim/Car/Landscape/Building/shader. That’s a huge coding challenge. It requires a genius team with drive and money.

It only takes a team of 3 people to download and label 10,000 meshes and the first AI mesh generator will be born. Then it will be an Avalanche. Have I gone two haddocks short of a banana?

2 Likes
2

Hah.

NOPE.

Look, we had “Strong AI is just around the corner!” in the past. Which was followed by AI Winter (1973, then 1988). No Strong AI in sight 50 years later.

Just because a tool can work in one type of medium, it does not mean it will be able to work in other mediums as well. Meshes include topologic information, are very different from images, and there are fewer of them.

Ten million meshes is not enough for training data. You need billions of examples or more. This can be done with images, but very hard to do with anything else.

Additionally there’s a matter of RPGMaker effect.

For example, Midjourney style is recognizeable, and as a result, over the time it may become associated with poor quality low budget work.

We already can generate billions of different trees. I can do that on my PC.

8412087--1111710--upload_2022-9-3_12-6-19.png

^ I can press a button and generate ANY number of those. They’ll be all different

I have local install of stable diffusion, therefore I have infinity of trees. Infinity of spaceships, of cities, of monsters, of people. Of almost anything I can think of. Well, it can’t draw driders, but I suppose I’ll have to live with that.

The thing is, It is not helpful.

With infinite amount of art material what happens is that you’ll end up getting lost and overwhelmed with the infinity of infinite possibilities.

Additionally, if technology becomes available, it does not mean it will be available to you or indies.

For example:
“A neural net is made capable of generating movies.
It requires 10 exaflops of computing power, 1 petabyte of GPU memory, 50 exabytes of storage, 1 exaflop of computing power and a 10 kiloqubit coprocessor”

You will not be running anything like that.

2 Likes
3

Well … yes and no. :slight_smile:

There are companies springing up that get funded, trying to to AI driven mesh generation. I recently came across Sloyd which left me flabbergasted - they got money FOR THAT?? :hushed:
But yeah, their current technology seems to be just so they have something to show, and something AI based is being developed. I hope it’s going to be 100 times better in visual quality, otherwise they are well advised to just take the money and run. :smile:

Unity has its ArtEngine … or rather: had. Now it’s supposed to move into the cloud or whatever. Haven’t seen much of it, but it reminded me of what Substance has already available.

You can find more examples, companies and individuals are looking into procedural and/or AI driven mesh generation, or content generation in general. Predicting the future at this point is going to be a game of chance at around 1-2 years into the future, nobody knows.

I believe what we WILL see in the near future is more generation of run-of-the-mill assets. Like today people actually charge $5 or so for a collection of low-poly rocks. Seriously? In a short time, those kind of environment assets will be entirely generated. Press a button and you get a hundred of these matching your criteria. Especially since those low-poly games are also tile-based and often created with a minimal budget. That is a niche I’m trying to focus on with Mesh Graph .

You could generate a lot already even today with Blender’s geometry nodes, but then low-poly art is just too easy to make so the effort into making a generally usable node tree making rocks or related assets seems like not a good tradeoff. Whereas for Node crafters it’s just too many assets to cover, hard to re-use a chair generator for tables already but what if you needed to make swords, too? And then trees, torches, coins, barrels, crates, bowls, shields, shelves, beds, doors, walls, towers, drawbridges, cars, trains, planes, ships, tanks, caterpillars, harvesters, traffic signs, etc etc etc - where do you start?

But we’re getting very close to where a lot of the low-poly art is just going to get generated, or at least generated and then manually fine-tuned by an artist. And it already is, I mean take a cube, add a minimal bevel, drag some vertices, and you got yourself a nice low-poly brick with most of the work done procedurally.

Making meshes is eventually going to be more like making textures, which we just take from some ubiquitous source and then use it to generate something more to our liking, modifying them procedurally. Artists aren’t going in the wild anymore, making pictures of all kinds of things, trying to make good textures. Only those involved in photogrammetry do that these days.

Meshes will be similar, you’ll have your basics that are modifiable, but if you want a demon face on that sci-fi wall you’re still going to have to merge those two manually in a way that works well within the art style.

We will see more parameterized generators, buildings are already quite common but often severely limited (ie only skyscrapers with limited variation in design). But generally architecture seems to be the env stuff that can be most easily turned into an algorithm.

But it’s much harder to make the generated meshes be useful for specific games, which is why this hasn’t taken off that much. You may be able to generate houses, but what if the game is supposed to have you explore the inside of those? You may be able to generate all sorts of bridges, but are they going to fit the low-poly style of the landscape?

Individual assets aren’t that hard to generate. Making them work in various games and art styles is non-trivial. So it’s best to focus on very specific, narrow-focus generators. SpeedTree comes to mind.

You’re bananas for thinking that 2D sprites, tiles, etc. will be at the forefront of generated assets. We would have that already if it were trivial. I cannot back it up with an argument right now. It just seems a lot easier to generate usable 3D topography than it is to generate 2D pixel art (not: textures used for 3D models). Maybe that’s because each pixel contributes a lot to the art style, so it really matters how you sprinkle them.

1 Like
4

Oh, and for the record, midjourney does not understand natural languages.

Your prompt is a meaningless gibberish for midjourney, it hallucinates you a result. It does not understand it.

Regarding neural networks, there’s already a problem where you can’t easily GET into that field and make something.

For example, some of currently existing image generator models can take 20000+ days of training on Tesla v100.
If you run the math with, say, google cloud prices, you realize that it will cost you between half a million and million USD to train one model like that.

This is already not affordable to many. You also often need fairly expensive hardware to run something like that, or you’ll be paying subscription for very limited number of results.

In essence, it is much more likely that we’ll end up in a situation that power neural network toys will be available exclusively to large corporations that can afford it, and it will be forever out of reach to indies which will not be have enough money or computing power. So rather than shortening the gap, it will widen it. And instead empowring indies to overthrow hollywood, it can empower hollywood instead.

1 Like
5

Midjourney can do that.
[AI art for games]( https://discussions.unity.com/t/890648 page-4#post-8408280)

I tried it with stable diffusion, it is very hard to do, because by default pixelart often morphs into mosaic and loses that pixelart feel.

8412135--1111737--upload_2022-9-3_12-42-13.png

But.

There’s already a project called Pixel Art Diffusion which supposedly can do this sort of stuff:


1 Like
6

Sprites can be photographic quality, cartoon styles and disney type cinematic quality. Running and jumping are fairly straightforward info within sprites that an AI can learn easier than going through billions of images from Dalle2.

The CEO of MidJourney says that his model takes about $50k to train once for all the images, so for a collection of 1 million sprite images, you’d be talking about $500 today and about 50 dollars in a few years time. He also says that 30fps movies are only 10 years away for those that have the money.

Mesh generation is just starting out… Nvidia has published models that will convert some photographs of a plant or a statue into a 3D mesh in about 10 seconds of processing time.

7

It seems natural that in the end there will be a Star Trek level computer with some sort of holodeck or VR device and you can dictate with a few sentences what kind of game you’d like to play and we will laught at the dark ages when somebody had to program and model things manually. Game developers would be obsolete at that point, of course. But is it 10-20 years or more like 30-50 years? Right now I’m thinking 30-50 but it is very hard to predict. There are still pieces missing like I don’t think it is clear how intelligent AI can get with current methods or if you can actually call it intelligence. It seems more like a specialized tool at this point.

1 Like
8

Polygons are a poor format for procedural/AI generation. SDFs are much better for 3d shape gen with compact formulas

1 Like
9

Hey Andyz, Cheers for the tip about SDF AI research. SDF are types of ISOsurface which U3D coders use for marching cubes. SDF are also used for 3D fractals like mandelbox using raytracing so they can indeed run in realtime at high resolutions. I didn’t know what they were and then I was like… Oh year I coded like 1250 SDFs

Deep learning model compute power was doubling every 100 days from 2010 to 2020, as opposed to Moore’s law which is doubling ever 2 years.

Nvidia’s A100 GPU has 58 billion transistors, 80GB RAM, and costs $15,000.
The 3090 has 28 billion and 24GB RAM, and costs about $1,200…

3090 is very slow for FP32, and the pro workstation premium is about 5X higher price per transistor, can see that reaching parity when AI acceleration becomes mainstraim in CPU’s.

AI accelerators are getting billions of dollars of research and companies like NVIDIA are studying 3D models in AI avidly.

It’s inevitable that AI-mesh for games will become very high quality.

10

Have you seen Unity Visual Search (blog post)? It’s limited to existing models from the asset store, obviously not generating anything. Unity should have integrated and expanded upon it.

11

Also thedre is a big push to make the training more efficient and less taxing on hardware. Ojne big bottleneck is the transformer who are O(n²) with n as sequence length, so people have been trying to compress the sequence length all while trying to increase the increase the sequence range, long former, reformer and all of the varient are trying to do just that, who know when we will achieve a breakthrough. Then there is also micro optimization to feed the hardware continuously, and then architecture tweak to “jump” through generation/training cycle faster.

1 Like