At the recent SIGGRAPH Nvidia have been talking about using AI to improve aspects of games and researching ways it could be used to bring ray tracing to games.
Ray tracing is a very brute force approach to lighting a scene realistically, would an AI system be able to dynamically select rays depending on scenery to reduce ray counts or use other types of tracing in areas of the scene that don’t need as much detail.
So could Unity’s new AI team work with the lighting team to see if AI could make light baking faster?
Someone will chime in with minor bickering because it is your thread and it will get locked while the foul mouthed Jim Sterling gets free run here but is PG-18 in some of his video comments.
Seems to be a smart enough idea. I’m curious what makes this AI as opposed to what makes this just a better algorithm. But we had that discussion on another thread.
I’d imagine they are already. The real time lighting world can’t be that big. It wouldn’t surprise me if they all know each other.
I’m fascinated by AI and how we might end up using it, so posted it here. The funny thing is we don’t really use good AI in games as it would be like setting the players up against IBMS Deep Blue and the players losing.
I would not have imagined that it would be useful in Ray Tracing, but in hindsight any algorithm that uses a brute force approach could probably be improved with AI as long as it has some kind of ‘lossiness’ or ‘overdraw’ where it works too hard for a given result e.g. ray tracing a wall.
On a side note would an AI be good for managing Unity’s job/threading system. For instance it could learn the game or even a players style and therefore could better manage a games CPU and GPU load across multiple cores.
I am doing a great deal of technical specs and 3D viz work for the AEC industries. I can see the use of a learning AI for optimization, calculating values for stress or air volume through ducts, water flow through a plumbing network and similar by using the SmartBIM metaData. Be nice to be able to train it through the Wolfram database as well. Another item I have to do research and dev for coming up is a 3D data visualization for AEC applications. AI can learn through associations and use requests for data manipulation, retrieval and storage to weight methods for display and access.
At some point the raytracing is going to become realtime and very good with powerful hardware behind it. But I think it may take time for industry wide adoption because so much time has already been spent massively optimizing the existing pipeline ie triangle-based rasterization. I think it has a disadvantage because of how expensive it it. If it were able to produce the same or better results on the same hardware it’d be instantly adopted widely.
I don’t care about what’s name AI, but I’m curious, what would be the input and what would be the output.
The other thing is how much they can compress a given DNN to be run in real time for game purpose.
thinking aloud about it
Raytracing typically works with triangle soups and ray intersection, the cost come from the exponential creation of secondary rays with bounced light, not the complexity of the algorithm (100 lines bro, triangles/ray intersection and you are go). In some way, it’s also about “path finding”, from the light to the screen.
Most of the optimization is really about decreasing the amount of unnecessary rays, those that don’t contribute to the final image, or compensating for ray that where needed but didn’t make it. Statistical method are popular because they allow reconstruction of the final data based on a small sampling (monte carlo based method and all).
The way I would experiment is basically like doing shadowmapping, using a “light view” to see the primary receiver and the “camera view” to see the “final caster”. Also I would voxelize the space, and feed these three mapping (really just baked into the voxel model) to a DNN to figure out the final image.
But I’m not sure that’s necessary because there is a precedent:
And how the inputs affected the outputs. Does it make decisions about manipulating input to output variables in an evolving manner that allows honing in on a solution to the current problem. Or is is a strict this equals that after manipulation output. This to me seems the base criteria for distinction between AI algos and regular algos.
You just had to go ahead and ruin otherwise informative post with another “What if” sales pitch.
The article specifies that visualization detals for preview purposes only.
Anyway, I think the best idea would be to let NVidia do that and then utilize their tech. They are interested in developing tech that utilizes their GPU, and they’re also responsible for acquisition of PhysX. I rememebr that Unreal heavily used some of the 3rd party libraries developed by (I think) Intel, for the purposes of raycasting and distance field generation.This sounds like a more reasonable approach than trying to “build your own casino”
I’d argue that having to take multiple photos with the light source placed in different positions each time is the very definition of complex data. As with most things involving neural networks, it takes a good deal of training before you can get the results in that video.
Yeah I meant that from a user perspective, these are cheap to generate as data, they just take time, but they can be entirely automated (even with real world data assuming you have a set up on a studio).
People keep saying you need a great deal of data, sure, but then it’s done and the ROI is potentially interesting, I mean compare to training one person then paying him to do the same job. I don’t see that as a negative, just let it train at night and come back, even if takes “weeks”, as long it’s useful forever after done correctly.
Nvidia should have a tone of data from it’s AI driving R&D lidar point graphs and stereoscopic video feeds. Imagine the amount of data a self driving car would receive from just driving around the block.
If you ensure that data is good enough to build 3d models and render them then you have the video data to check the render for accuracy.
It’s kind of funny/poetic in a way as Nvidia train an AI to see the world they also start to realise that they can reverse the process and train an AI to render the world.
I am guessing that they use the well known ability of neural nets to learn edge detection and had it learn a specific type of edge detection that would typically define detail. How much a brightness gradient shifts over what distance, how much contrast between this pixel and the next and the next to that etc…, how much a color differentiates from pixel to pixel or pixel grid cell average to pixel grid cell average. It then determines where to cast the rays depending on the density of edge detection in an area of a given size. This raytracing selection could also be trained by having it continually check it’s results against an unbiased render of the same scene and weight the methods most closely reproducing the unbiased render.
Neural network are currently absolutely revolutionizing image treatment and procesing, that’s one thing where it’s not academic exercise anymore and understood enough that little joe in his garage can do it on mom’s computer. Has any people seen the image scaler from old trained NN (vgg)? They are all over github. It’s entirely possible that future consoles (maybe not immediate) could use NN to enhance graphics if it’s fast enough. Gave low quality CG image as input, get photoreal as output.
Ok… that one isn’t revolutionary in application yet, but it is pretty nifty. Also, it combines neural networking with crowdsourcing which is potentially really cool.
For the artistically challenged, I can see this being expended with a text output to create a digital art director, it would tell what’s wrong, what to correct and how. We already now that DNN can match image with sentence template easily. https://arxiv.org/pdf/1707.03981.pdf https://arxiv.org/pdf/1707.05251.pdf
The next practical thing on the short term (after more efficient style transfer) is super resolution and various image enhancement, I think right now they resolve image in 0.8 second, but as far as I know it’s generic implementation for broad usage, might be able to have 10x improvement on proper optimization. https://arxiv.org/pdf/1707.02921.pdf https://arxiv.org/pdf/1707.02880.pdf
Image synthesis is lagging behind a bit, but we will eventually get there too:
