I was reading up more on IK and Animation Rigging packages, to enable combination of live recordings with picking up objects in a scene (or putting hand on a desk or door knob etc where you need precise positioning control). I assume you either have to live capture the exactly correct positions to line up with the real and virtual worlds, or you need to override inside Unity to make adjustments (e.g. using IK) and then blend between the recording and the IK.
There is OnAnimateIK() where you can script up your own IK. But you have to use it with an Animator Controller and set “IK Pass” to true. See Ik in Timeline animations
Using Animator Controller however appears not to work well with override tracks according to Blending between timeline and animation controller . So you have to choose between Animation Controllers or Timelines with animation clips with overrides and Avatar Masks. Unless this has changed, they don’t play well together. I use the override tracks for facial expressions and hand positions (because I don’t have reliable finger tracking)… hence I don’t think using an Animator Controller will work for me.
There is the Unity Animation Rigging package that allows you, in timeline, drag the hand etc as an IK. I like this video overview
from Unity. (Also Animation Rigging and Timeline Quick Guide ) You can use Animation Rigging in Preview mode, which sounds good for use with Sequences. You animate the weight up where the adjustment to the recorded movement is needed, then animate it down afterwards. So as long as the actor got “pretty close” to where the object is, you are set. Note: the video uses Animator Controller from what I can see.
But Animation Rigging and Sequences has been giving me pain! E.g. Bug? Possible conflict between RigBuilder and sequences . More example exceptions in Console:
MissingComponentException: There is no 'PlayableDirector' attached to the "interior" game object, but a script is trying to access it.
You probably need to add a PlayableDirector to the game object "interior". Or your script needs to check if the component is attached before using it.
UnityEngine.Playables.PlayableDirector.get_playableAsset () (at <b063ef8cb96c4883b75f10561b7f0a02>:0)
UnityEditor.Timeline.SequenceHierarchy.ValidateSubElements (System.Collections.Generic.List`1[T] subElements, UnityEngine.Playables.PlayableDirector director, System.Collections.Generic.List`1[UnityEditor.Timeline.SequenceHierarchy+SequenceBuildingBlock]& buildingBlocks) (at Library/PackageCache/com.unity.timeline@1.6.1/Editor/State/SequenceHierarchy.cs:237)
UnityEditor.Timeline.SequenceHierarchy.FromSequencePath (UnityEditor.Timeline.SequencePath path, System.Boolean forceRebuild) (at Library/PackageCache/com.unity.timeline@1.6.1/Editor/State/SequenceHierarchy.cs:188)
UnityEditor.Timeline.WindowState.SetCurrentSequencePath (UnityEditor.Timeline.SequencePath path, System.Boolean forceRebuild) (at Library/PackageCache/com.unity.timeline@1.6.1/Editor/State/WindowState.cs:441)
UnityEditor.Timeline.TimelineWindow.RestoreLastSelection (System.Boolean forceRebuild) (at Library/PackageCache/com.unity.timeline@1.6.1/Editor/Window/TimelineWindow_Selection.cs:65)
UnityEditor.Timeline.TimelineWindow.RefreshSelection (System.Boolean forceRebuild) (at Library/PackageCache/com.unity.timeline@1.6.1/Editor/Window/TimelineWindow_Selection.cs:60)
UnityEditor.Timeline.TimelineWindow.OnEnable () (at Library/PackageCache/com.unity.timeline@1.6.1/Editor/Window/TimelineWindow.cs:127)
UnityEditorInternal.InternalEditorUtility:LoadSerializedFileAndForget(String)
UnityEditor.WindowLayout:LoadDefaultWindowPreferences()
and
InvalidOperationException: Sequence contains no elements
System.Linq.Enumerable.Last[TSource] (System.Collections.Generic.IEnumerable`1[T] source) (at <b024aa277ab0422080e4a1dbe1f3c835>:0)
UnityEditor.Graphs.AnimatorControllerTool.UpdateStateMachineSelection () (at <ac9a6525061f4e29af6bedf66fd7c5fe>:0)
UnityEditor.Graphs.AnimatorControllerTool.RebuildGraph (System.Boolean updateSelection) (at <ac9a6525061f4e29af6bedf66fd7c5fe>:0)
UnityEditor.Graphs.AnimatorControllerTool.UndoRedoPerformed () (at <ac9a6525061f4e29af6bedf66fd7c5fe>:0)
UnityEditor.Undo.Internal_CallUndoRedoPerformed () (at <b0d21c8593a94bc3a2b5da97f0d5796a>:0)
UnityEditor.EditorApplication:Internal_CallGlobalEventHandler()
and
NullReferenceException: Object reference not set to an instance of an object
UnityEngine.Animations.Rigging.RigBuilder.UpdatePreviewGraph (UnityEngine.Playables.PlayableGraph graph) (at Library/PackageCache/com.unity.animation.rigging@1.1.1/Runtime/AnimationRig/RigBuilder.cs:178)
UnityEngine.Timeline.AnimationPreviewUpdateCallback.Evaluate () (at Library/PackageCache/com.unity.timeline@1.6.1/Runtime/Animation/AnimationPreviewUpdateCallback.cs:40)
UnityEngine.Timeline.TimelinePlayable.Evaluate (UnityEngine.Playables.Playable playable, UnityEngine.Playables.FrameData frameData) (at Library/PackageCache/com.unity.timeline@1.6.1/Runtime/TimelinePlayable.cs:288)
UnityEngine.Timeline.TimelinePlayable.PrepareFrame (UnityEngine.Playables.Playable playable, UnityEngine.Playables.FrameData info) (at Library/PackageCache/com.unity.timeline@1.6.1/Runtime/TimelinePlayable.cs:249)
UnityEngine.Timeline.DirectorControlPlayable:ProcessFrame(Playable, FrameData, Object) (at Library/PackageCache/com.unity.timeline@1.6.1/Runtime/Playables/DirectorControlPlayable.cs:123)
When I move the IK target, nothing happens (the arm does not bend) in Preview or Record mode.
Further, if I create a Timeline outside of Sequences and Copy / Paste the character from the Sequence timeline into my stand-alone timeline, Rigging Animation works - I can do IK movement. So sounds like a conflict between the Sequences package and Animation Rigging causing the above stack traces, which is disabling the functionality. (Does that seem reasonable?)