Unity AOV Recorder update

marief_unity · July 26, 2019, 5:38pm

Hey there !

AOVRecorder 0.1.0-preview-3 has been officially released ! This is a Recorder plugin that relies on High Definition Render pipeline.

AOVs (Arbitrary output variables) - sometimes called render passes or render layers - are data that are being calculated to produce the beauty pass (=the final render).
AOVs examples: depth, normals, albedo…
Being able to export them separately allows more options at compositing step.

Please note, this is a first preview version (beta). There are known issues and limited settings for now :

First frame (0) sometimes is black
Active Camera target is not supported by HDRP
Depth data is incorrect if the scene distance is large with HDRP version < 6.8
PNG and JPEG files are written as linear file instead of sRGB.
Simultaneous recording of the beauty pass and other render passes if TAA is enabled will generate artifacts
Recorder issue: Timeline recording of the beauty pass will record black with HDRP

Also : it is highly recommended to use this plugin with the latest version of HDRP possible.

See documentation.

To install it, open Package Manager window, make sure you enable “show preview packages” advanced option and install com.unity.aovrecorder.

Please let us know how testing goes. Any feedback is more than welcome

Thanks !

Mad_Banny · August 1, 2019, 5:17pm

Hi!
Here is my feedback:

Unity 2019.2, HDRP 6.9.1

NullReferenceException: Object reference not set to an instance of an object
UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.EnableAOVCapture (UnityEditor.Recorder.RecordingSession session, UnityEngine.Camera cam) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:43)
UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.NewFrameStarting (UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:138)
UnityEditor.Recorder.Recorder.SignalInputsOfStage (UnityEditor.Recorder.ERecordingSessionStage stage, UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/Recorder.cs:160)
UnityEditor.Recorder.RecordingSession.PrepareNewFrame () (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/RecordingSession.cs:184)
UnityEditor.Recorder.RecorderComponent:Update()

NullReferenceException: Object reference not set to an instance of an object
UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.DisableAOVCapture (UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:127)
UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.FrameDone (UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:150)
UnityEditor.Recorder.Recorder.SignalInputsOfStage (UnityEditor.Recorder.ERecordingSessionStage stage, UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/Recorder.cs:168)
UnityEditor.Recorder.RecordingSession.RecordFrame () (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/RecordingSession.cs:134)

4811771–460679–HQ_Deffered_HDRenderPipelineAsset.zip (1.71 KB)

jean_unity_film_tv · August 1, 2019, 5:32pm

Hi Mad_Banny,

Sorry to see that you are having problems.
Here are a few questions for trying to isolate the issue.
Are you using the AOVRecorder with an HDRP scene? The AOV Recorder will only record AOVs from an HDRP scene.
How do you start the recording? With the recorder window or with a record clip on the timeline?
What are your recording options?
What is your camera input? Are you trying to record the main camera or a tagged camera?

Would it be possible to zip and upload you project for us to investigate and try to reproduce the issue here?
Thanks,

Jean

Mad_Banny · August 2, 2019, 8:55am

Yes, I use it with HDRP scene.
I tried both window and timeline.
I tried all recording options.
I guess this is due to RP asset settings

Mad_Banny · August 2, 2019, 8:57am

Tried it in the default scene and it worked

jean_unity_film_tv · August 2, 2019, 1:17pm

Glad to see that it worked for you with the default scene, I will try it here with your render pipeline asset to see if I can reproduce your issue.

jean_unity_film_tv · August 2, 2019, 2:03pm

Hi Mad_Banny,
To give you an update, I wasn’t able to reproduce your issues with the AOV Recorder using the HQ_Deffered_HDRenderPipelineAsset that you provided. If you can provide a small project that would reproduce the problems or a list of steps that would be very useful.

Thanks,

Jean

mahenkumar · August 9, 2019, 10:14am

I can’t find this option in my Unity 2019.2. I am using all the required versions.
Is it only for pro Unity users ?

marief_unity · August 12, 2019, 4:25pm

Hey @mahenkumar !

Did you install the package via package manager ?

If so, do you have errors in the console ?

Thanks !

ps: AOV Recorder is available for everyone.

xCyborg · August 16, 2019, 1:12am

I do hope you intend to quickly integrate that with Unity Recorder, also to extend support of AOVs to LWRP and even standard pipeline.
Please tell me you don’t intend to keep it separate.

marief_unity · August 16, 2019, 1:33pm

Hey @xCyborg

AOVRecorder is a plugin of Recorder, so it is actually part of Recorder. We choose to make it a separate package because it highly depends on High Definition RP (it’s using HDRP new AOV Request API) and we didn’t want this dependency to burden Recorder, there are a lot of users that have no interest in AOVs.

For now, there is no AOV API in LWRP, but when there will be, the plan is to indeed allow AOVRecorder to use it.

There is no plan to make AOV Recorder work with Legacy Render Pipeline.

xCyborg · August 17, 2019, 1:27am

@marief_unity Well, as far as I’m concerned, it’s another fragmented duplicate package.

andrzej_1 · September 28, 2019, 10:37am

Could we at least move the option to add AOV to the import window of the Recorder (just like samples) with a huge !warning! text about HDRP support only? Making it a separate package (but still the same tool and GUI element to interact with), seems a bit redundant.
One thing that would be very useful for anyone working in video production would be to have ‘masks’, similar to cryptomattes or multimasks? Color coding by material/entity/layer … This is a must for compositing.

marief_unity · September 30, 2019, 8:26pm

Hey @andrzej_1 ,

AOVRecorder as a separate package :

I understand your concern but we made a choice considering :

AOVRecorder is a tool that not everyone is interested in
HDRP is a required dependency BUT we can’t impose that dependency to all Recorder users
Package management is still in development, for example conditional dependency remains a too recent feature (we support until 2018.4)
And we know by experience (including in Recorder) using reflection causes more issues in the long term.

That said, 2 things will happen that could be disruptive of this decision:

AOV Request API being ported in URP (no ETA)
2018.4 being “deprecated” enough so we can freeze the Recorder dev on it (not happening in a near future, but will happen for sure)

Until that, it is what it is. We remain confident we chose the best solution possible considering the current parameters, even with its obvious disadvantages.

Cryptomatte :
Adding ObjectID / Cryptomatte passes is indeed in our scope. I can’t provide you an ETA though.

Thanks for your feedback !

Solovykh · October 5, 2019, 12:54am

Hey, I’m trying to write my own custom recorder. However, all the recorders in the Recorder packages are marked as “internal”. How did you guys manage to have AOVRecorder inherit from BaseTextureRecorder which is internal?

cv_dg · October 7, 2019, 11:07pm

We are trying to use the AOVRecorderSettings in the scripted capture method. The ImageRecorderSettings work just fine for us, but the one thing we can’t seem to figure out is where to set the AOVGType in the settings. We want to be able to record out the alpha for a matte. Does anyone know how to do this?

Unity 2019.1.9f1
Unity AOV Recorder preview.5 - 0.1.0

cv_dg · October 7, 2019, 11:30pm

Looks like this is the AOVGSelection which is internal and defaults to AOVGType.Beauty.

cv_dg · October 8, 2019, 12:03am

The other thing with this is even though I set the AOVCameraInputSettings, it isn’t setting the targetCamera in AOVCameraInput.RenderAndReadbackAOVCapture.

aovRecorder.imageInputSettings = new AOVCameraInputSettings
{
    cameraTag = "MainCamera",
    source = ImageSource.MainCamera,
    outputWidth = 384,
    outputHeight = 216,
    captureUI = false
};

So I had to force it to targetCamera = Camera.main so that that it would let me run without a nullreferenceexception.

if (aovRecorderSettings != null)
{
    targetCamera = Camera.main; // Added to stop NullReferenceException
    targetCamera.Render();
    RenderTexture.active = outputRT;
    readbackTexture.ReadPixels(new Rect(0, 0, outputWidth, outputHeight), 0, 0, false);
    readbackTexture.Apply();
    RenderTexture.active = null;
}

I am also seeing that the first frame is always missing as shown in the Known Issues log.

dschmuck · October 15, 2019, 9:08am

Hi!
Really great that you guys added this package. We are using Unity to render and still do a traditional composting workflow as a final step after unity, so any additional passes we can get out of unity are greatly appreciated!

We still have a bunch of questions, we hope you can help us with!

will multichannel exrs be eventually supported? For our comp workflows it just makes everything more conveniant
currently there only seem to be the commonMaterialProperties available as render passes (plus depth)
It would be fantastic to have access to the other information you can view in the debug view like the LightingModes for example to get proper diffuse lighting and specular lighting passes.
I tried to expand the current package to be able to render these Lighting passes by using “LightingProperty” and “SetDebugLightingMode”, but I couldn’t quite get it to run (it alway rendered the beauty instead of the expected lighting pass)
will MotionVector pass be included soon? we want to add motion Blur in Comp!
How would you go about adding more complex custom render passes?
I am currently implementing an object/material/asset ID solution that depends on shader overrides. Currently it is done using an additional render process after rendering beauty and the other passes.
I would love to include this process directly into the “main” render process and render it to its own pass.
Do you have any suggestions on how to get that to work?
Lastly the ID passes rely on metadata that should be written into the header of the exr. Currently this is done in a separate follow-up process. Is it somehow possible to write medata into the exrs when rendering with aov recorder or are you guys thinking about adding that?

I hope you guys can give us some answers.
Thanks for all the good work for making films with unity!!

jean_unity_film_tv · October 29, 2019, 11:40am

Hi @dschmuck ,

See my answers below after >>>

Hi!
Really great that you guys added this package. We are using Unity to render and still do a traditional composting workflow as a final step after unity, so any additional passes we can get out of unity are greatly appreciated!
We still have a bunch of questions, we hope you can help us with!
will multichannel exrs be eventually supported? For our comp workflows it just makes everything more conveniant

Yes we are planning to add multichannels exr support to the AOV Recorder.

currently there only seem to be the commonMaterialProperties available as render passes (plus depth)
It would be fantastic to have access to the other information you can view in the debug view like the LightingModes for example to get proper diffuse lighting and specular lighting passes.
I tried to expand the current package to be able to render these Lighting passes by using “LightingProperty” and “SetDebugLightingMode”, but I couldn’t quite get it to run (it alway rendered the beauty instead of the expected lighting pass)
will MotionVector pass be included soon? we want to add motion Blur in Comp!

Also, yes we are planning to make more AOVs types available like the motion vector pass.
In HDRP version < 6.7 the AOV recorder is using the debug api for creating the AOVs, from HDRP version 6.7 and above we are using a new AOVRequest API. This allow us to make requests for the AOVs that we want to gather and record.

How would you go about adding more complex custom render passes?
I am currently implementing an object/material/asset ID solution that depends on shader overrides. Currently it is done using an additional render process after rendering beauty and the other passes.
I would love to include this process directly into the “main” render process and render it to its own pass.
Do you have any suggestions on how to get that to work?

To create an object ID pass using the new HDRP Custom Pass might allow you to do what you want.
Redirecting to latest version of com.unity.render-pipelines.high-definition

Lastly the ID passes rely on metadata that should be written into the header of the exr. Currently this is done in a separate follow-up process. Is it somehow possible to write medata into the exrs when rendering with aov recorder or are you guys thinking about adding that?
I hope you guys can give us some answers.

Currently this is not possible. But I can see how this could be very useful.

Thanks for all the good work for making films with unity!!

You are welcome!

Topic		Replies	Views
AOV Recorder not exporting the passes Unity Engine Audio-Video , Question	2	914	September 7, 2021
Recording AOV passes with the Recorder 3.0 Unity Engine Audio-Video , Question	8	1408	June 10, 2021
Generate pixel-wise Object Labels. Unity Engine HDRP , com_unity_render-pipelines_high-definition	3	1080	September 10, 2019
Recorder release announcements Unity Engine Audio-Video , Official	12	11074	June 2, 2022
recorder only renders one iteration of the path tracing Unity Engine Audio-Video	7	2197	March 21, 2021

Unity AOV Recorder update

Related topics