Confusion around right approach to override Timeline animated properties by script

[fherbst]

I’m trying to wrap my head around something that I think can’t be that difficult, but I haven’t found an approach yet that works.

Here’s what I’m trying to do: I have an AnimationTrack that contains 15 different clips that all animate the same object. The clips have some overlap so that they blend into each other.
Now I want to be able to, per each of these “sections” (the clip relate to environmental effects for specific sections of an experience), to override what they do. E.g. one of the sections has “animate the skycolor to red”, but I want the user to be able, at runtime, to override that with green, while still having the proper fades to the previous and next clips.

First thing I tried was to use a custom script to figure out the right weights each clip has and to apply the right override color based on these weights. That does not work for blending between clips, as the “original” color will start to fade in and I have no way to mix “previous clip color” with “override color”.

Second thing I tried was to create replacement Animation Clips with custom keyframes and put them into the AnimationClipPlayables at runtime when needed. This does not work at runtime since there is currently no API for creating AnimationClips that are compatible to timeline that works outside of the Editor.

Third thing I tried was having a custom Mixer/Track (since, after all, I want to mix the clips in a custom way), but that failed since even with decompiling the Timeline dll most of the interesting stuff (“OnCreatePlayableGraph”) is internal and cannot be overriden, so I can’t replace the AnimationMixer with my custom one. The only thing I can override is “CreateTrackMixer”, and according to decompilation, that’s not even used by the AnimationTrack class! It’s hardcoded to use AnimationMixerPlayable! … that would’ve been the right spot if not for that internal oversight.

Fourth thing I tried was doing it the “timeline way” and override the actual graph that is generated.
Currently it looks like this:

With the override (modifying the graph in PlayableDirector.played), it looks like this:

However, the issue with this approach is that the Timeline doesn’t set the weights on these three AnimationMixers, it still changes the weights of the clips. So, no luck again.

Fifth Thing was looking into the experimental AnimationStream. From what I found, it seems that these can be used in AnimationStreamPlayables that can modify the data running through them. This means I’m running into the same issue as with approach #4, of the Timeline not being able to set weights on something else than the actual AnimationClip unless I hack that in as well.

Any advice on what the right plan of action would be here? I guess the base question boils down to:

“How do I modify the output of an AnimationClip before it goes into an AnimationMixer, while still using the Timeline?”

[seant_unity]

I don’t think what you are trying to do is possible using timeline and built-in animation tracks. Based on the description of the problem, it seems like you would be better off writing a custom track and mixer to do this, at least for the dynamic elements that you are trying to animate.

[fherbst]

These dynamic elements have animationclips with dozens of artist-edited parameters and nicely animated motion curves. The idea was just to allow the user to modify some of these, and only when he wants to. Well, a temporary “override” in exactly that sense.

Is there at least some documentation around or some example how to write a custom track and mixer that can also take AnimationClips and blend them? Like I said, I thought the “best” way to do this would be to just subclass the AnimationTrack and the AnimationMixer and change what’s needed, but that doesn’t seem possible as of now.

[seant_unity]

There may be more thorough examples I am not aware of, but here’s a basic animation track and clip you can start from.

Track:

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Playables;
using UnityEngine.Timeline;
using UnityEngine.Animations;

namespace Custom
{
    [System.Serializable]
    [TrackClipType(typeof(CustomAnimationAsset))]
    [TrackBindingType(typeof(Animator))]
    public class CustomAnimationTrack : TrackAsset
    {
        public override IEnumerable<PlayableBinding> outputs
        {
            get { yield return new PlayableBinding { sourceObject = this, streamName = name, streamType = DataStreamType.Animation, sourceBindingType = typeof(Animator) }; }
        }

        public override Playable CreateTrackMixer(PlayableGraph graph, GameObject go, int inputCount)
        {
            return AnimationLayerMixerPlayable.Create(graph, inputCount);
        }
    }
}

and the clip:

using System;
using UnityEngine;
using UnityEngine.Animations;
using UnityEngine.Playables;
using UnityEngine.Timeline;

namespace Custom
{
    [Serializable]
    public class CustomAnimationAsset : PlayableAsset, IPropertyPreview
    {
        [SerializeField] AnimationClip m_Clip;

        public override Playable CreatePlayable(PlayableGraph graph, GameObject go)
        {
            // could create a subgraph of script playables, mixers, etc...as well. Use playable.SetPropogateSetTime()
            // on any non-leafs nodes to have time values set by timeline propagate down the graph automatically
           
            return AnimationClipPlayable.Create(graph, m_Clip);
        }

        // Adds the animated values to 'preview' mode
        public void GatherProperties(PlayableDirector director, IPropertyCollector driver)
        {
            if (m_Clip != null)
            {
                driver.AddFromClip(m_Clip);
            }
        }

    }
}

I hope that helps you solve the problem!

[fherbst]

Hey there, I’m stuck again! I tried today getting a custom graph to work using your example code. After fixing some stuff that is actually already obsolete in 2018.2, I ended up with this:

[System.Serializable]
    public class OverrideBehaviour : PlayableBehaviour
    {
        public OverrideContent content;
        public AnimationMixerPlayable mixer;

        [Header("Output")]
        public float mixerWeight;

        public override void PrepareFrame(Playable playable, FrameData info)
        {
            Debug.Log("[Prepare Frame] called every frame, weights: " + info.weight + " - " + info.effectiveWeight);
            base.PrepareFrame(playable, info);
        }

        public override void OnBehaviourPlay(Playable playable, FrameData info)
        {
            Debug.Log("[OnBehaviourPlay] starts playing with weight : " + info.weight + " - " + info.effectiveWeight);
            base.OnBehaviourPlay(playable, info);
        }

        public override void ProcessFrame(Playable playable, FrameData info, object playerData)
        {
            Debug.Log("[ProcessFrame] this is never called");
            base.ProcessFrame(playable, info, playerData);
        }
    }

[Serializable]
    public class OverrideAnimationAsset : PlayableAsset, IPropertyPreview
    {
        [SerializeField] AnimationClip m_Clip;
        [SerializeField] OverrideContent overrideContent;

        public override Playable CreatePlayable(PlayableGraph graph, GameObject go)
        {
            // could create a subgraph of script playables, mixers, etc...as well. Use playable.SetPropogateSetTime()
            // on any non-leafs nodes to have time values set by timeline propagate down the graph automatically
           
            var p = AnimationClipPlayable.Create(graph, m_Clip);
            var newPlayable = AnimationMixerPlayable.Create(graph, 0, false);

            var overridePlayable = ScriptPlayable<OverrideBehaviour>.Create(graph);
           
            var overrideBehaviour = overridePlayable.GetBehaviour();
            overrideBehaviour.content = overrideContent;
            overrideBehaviour.mixer = newPlayable;

            // return overridePlayable;

            newPlayable.AddInput(p, 0, 1f);
            newPlayable.AddInput(overridePlayable, 0, 1f);
            newPlayable.SetPropagateSetTime(true);
           
            return newPlayable;
        }

        // Adds the animated values to 'preview' mode
        public void GatherProperties(PlayableDirector director, IPropertyCollector driver)
        {
            if (m_Clip != null)
            {
                driver.AddFromClip(m_Clip);
            }
        }
    }

with OverrideContent being a ScriptableObject that’s supposed to hold my content.

This, together with your TrackAsset code, works for having the clips in the timeline, being able to set a ScriptableObject on it and seeing that the graph is right in the PlayableGraphVisualizer.

However, to my despair, OnProcessFrame is never called! Both in Play and Edit mode, OnPrepareFrame is happily called, telling me that the playable has a weight of 1, but OnProcessFrame - never ever. Am I doing anything wrong here? I have used custom Playables before, and there OnProcessFrame “just worked”.

I tried multiple things, including playing with the weights and the weight normalization, only having this custom playable in the mixer, … but it just won’t process the frame…

[seant_unity]

ProcessFrame is never called when the output is an animation playable output (i.e the stream type is animation). Most custom tracks are ‘script playables’ which uses an output type that (only) calls ProcessFrame, but animation doesn’t call process frame (which is the stage used to evaluate the animation).

What are you trying to using ProcessFrame for?

[fherbst]

Same as in the whole thread – I’m trying to override the output of several animation clips on a Timeline from code.
So what I did there is what you suggested, creating a custom track with clips that build a little Playable graph by themselves – basically a Mixer with two Inputs, the AnimationClipPlayable and my custom ScriptPlayable, both with weight 1 (weight normalization off). That looks fine in the GraphVisualizer.
My idea was that basically once the AnimationClipPlayable has processed the frame, the OverrideBehaviour can do custom stuff on top. If ProcessFrame isn’t the right place to do it, what’s the right method to override for this?

Adding to the options above, I tried AnimationJobs, but they aren’t documented at all and I couldn’t get them to work. I found one snippet in a Unity blogpost, but that is outdated and contains code that is already obsolete again / didn’t even make it into a public build.

[seant_unity]

Ah, ok… I understand now. My confusion was that I assumed you were trying to blend in different clips that targeted different properties, instead of dynamically trying to change the target property.

There are several solutions that are possible here. The first is editor side, generate all possible clips you might need. That maybe a bit crazy, but overriding/changing the bindings on an animation clip is not possible in the player. You could take an animation clip that animates color.r, and generate a new clip that animates color.g for example. Then have a script playable that just sets the weights to the appropriate clips. Obviously this could generate a lot of clips, depending on how many combinations you need.

The second is have a playable that extracts the AnimationCurves from an AnimationClip, then in PrepareFrame (or ProcessFrame) update the property on the binding directly. You could change your binding to a script playable output instead, but it would effectively be a manual animation system. You would likely need to use reflection to get the properties to animate.

The third, if you are willing to jump into the 2018.2 beta, is C# animation jobs. (You mentioned you tried animation jobs, maybe this will help). It looks like it might be what you are looking for. It’s still experimental, but it seems like you could put a C# job playable between the mixer and the clip playable.

Hopefully one of those suggestions works for you. The most flexible solution would be being able to retarget or create a new animation clip at runtime but that isn’t possible because the animation clips are optimized during the player build process, and become baked.

[Dr.Game]

HIHI~How can I set the Animation Clip (Override) at RunTime??
Is it only possible in custom track?
I want use TimeLine as a template.Want to feed character ,animationclip,specific position at running time ~~

[fherbst]

@Dr.Game

HIHI~How can I set the Animation Clip (Override) at RunTime??
Is it only possible in custom track?
I want use TimeLine as a template.Want to feed character ,animationclip,specific position at running time ~~

I would recommend making a new thread for this different issue.

[daxiongmao]

@fherbst Did you ever find a solution that worked for you?

[fherbst]

@daxiongmao yes indeed: GitHub - needle-tools/timeline-mixer: allows for overriding timeline animations per animator