Quality is decent and runs at ~30FPS without screen recording. Reflections and refractions showcased.
Video link - https://youtu.be/shGus4ggq6o
Here’s a comparison between Android and Blender cycles render.
Clearly Blender is better but for a mobile build, I think it’s pretty good.
I had already released a similar project in Godot previously and porting it over was pretty simple with Sentis. Created the scene and training data with Blender in a couple of hours and did the training overnight in Kaggle. Sentis is used for inference.
I have the apk file but I’m not sure if I’m allowed to share it here.
There are a couple of issues.
Build runs smoothly but crashes after 20-30 seconds. I get the following error statements.
In the video and screenshots of the game above, you’ll notice horizontal lines. Probably a minor issue but I’m not sure how to get rid of them.
Using Sentis has been pretty smooth and let me know if you have any questions or suggestions. Thanks!
2 Likes
Wow. That’s pretty amazing! So is the input like a depth map or something? I don’t even know how it works 
That looks awesome!
Looking at your log, it seems to fail when allocating cpu memory for a tensor.
My guess is that you’re running out of memory. so maybe forgot to dispose your tensors during execution?
A few questions:
- could you share the model (can be random weights) so we can try to repro it on mobile
or maybe the full code, but I understand that you might be uncomfortable with that
- you seem to be using CPU worker for what seems to be GPU task, may I ask the reason?
@yoonitee @alexandreribard_unity
The technique is specific to the scenes and objects. It is intentional and very fast to train. Took me 2 hours to train in my laptop RTX 3060 so a better PC/online notebooks could do it in less than an hour. The only input to the model is the camera position and the model is a cGAN trained with Blender renders of the same scene. There’s plenty of room for improvement and fine-tuning as I’ve only spent a 10 minutes making the model.
The code is pretty much copy pasted from Sentis docs.
private void Update()
{
float[] data = new float[] { (180 - mainCamera.eulerAngles.y) / 360f };
TensorFloat input = new(new TensorShape(1, 1, 1, 1), data);
worker.Execute(input);
var output = worker.PeekOutput() as TensorFloat;
nnMaterial.mainTexture = TextureConverter.ToTexture(output);
input.Dispose();
output.Dispose();
}
GPU worker only yielded 4 FPS on my phone. I randomly tried switching to CPU and performance drastically improved. I am confused by this as well. Are there any caveats to using CPU? Everything seems fine but I’m not sure.
2 Likes
Ah yes so here is the reason for your crash
input = new TensorFloat is never disposed and is allocated every frame, that’s a problem and will cause a out of memory after a while.
TextureConverter.ToTexture also creates a new texture every time it is called, it would be best to allocate it once and copy the result back into it
I see. I assumed
input.Dispose();
would dispose and it is done at the end too.
oh sorry missed that…
the it’s the
TextureConverter.ToTexture
that is the culprit
1 Like
I’m a bit confused to why the GPU would take so long to compute.
Do you mind sharing the model?
Here you go.
It’s a simple upscaling model with a bunch of convTranspose layers. I did some testing today and found that even with 3 layers and couple of hundred parameters, the game runs still runs at 4-5FPS on GPU which means it’s probably my fault and I’m missing something minor. I couldn’t figure it out but hope it will be useful to you.
it would be best to allocate it once and copy the result back into it
Could you give me pointers on how to achieve this? I tried different things (SetPixel, BlockCopy) and all of them are slow. TextureConverter.RenderToTexture also causes the crash.
TextureConverter.RenderToTexture(…)
you can pass in a pre-allocated texture aswell as your tensor. we’ll copy the values of the tensor into the RT
Yes, this is exactly what I’m doing but it still crashes 
Neither RenderTexture nor Texture2D works…
private RenderTexture outputRTTexture;
private void Start()
{
...
outputRTTexture = new(256, 256, 3);
}
private void Update()
{
float[] data = new float[] { (180 - mainCamera.eulerAngles.y) / 360f };
TensorFloat input = new(new TensorShape(1, 1, 1, 1), data);
worker.Execute(input);
TensorFloat output = worker.PeekOutput() as TensorFloat;
TextureConverter.RenderToTexture(output, outputRTTexture);
input.Dispose();
output.Dispose();
}
I am getting the expected output. It just crashes after a while and profiler shows increasing memory usage over time.
You should not dispose the output tensor in this case, as far as I remember.
Do you get the correct result in the render texture?
is it possible to disable Burst and test it?
Hi,
We’ve received your project. It’s currently being investigated and is known internally as issue 120.
1 Like
