Expecting too much from network movement interpolation?

[HiddenMonk]

I am working on interpolating movement and am displeased by the results I am getting.
The results are alright I guess? I think it isnt so great if I am spectating someone though…
(tested only on localhost so far)

Here is an example of how I am doing things.
Click for code

        float updateRate = 20;
        //...
 
        public void Update()
        {
            Vector3 prevPos = transform.position;

            currentInterpolateTime += (Time.deltaTime * updateRate);

            transform.position = Vector3.Lerp(startPosition, targetPosition, currentInterpolateTime);
           
            //Check to see if we ever move backwards.
            Debug.DrawRay(transform.position, (Vector3.up + ExtVector3.Direction(prevPos, transform.position)) * 2f, Color.yellow, 1f);
            //Check to see our movement consistency.
            Debug.DrawRay(transform.position, Vector3.up * (Vector3.Distance(prevPos, transform.position) / Time.deltaTime), Color.green, 2f);
        }
       
        //This is called before Update
        protected override void OnAfterDeserialize()
        {
            startPosition = transform.position;
            targetPosition = position.value;
            currentInterpolateTime = 0;
        }

Here is an image example
Note that I am just moving at a consistent rate on the server side, and this is the result on the client side.
Click for image

As you can see by the image, basically each time we receive an updated position packet, the distance is different, which seems to cause inconsistencies in the movement between each update position packet.
This may be due to the other side not exactly sending every so and so time since its running at 60fps and the sendrate is 20 times a second, so there can be a 16ms difference in distance etc…(remember, this is just localhost so far).

What I want to ask is, is this to be expected? Is it just not possible to really lerp smoothly across all movement update packets with the current info I am sending?
Is there anything I can do to improve this?

For example, I see people talk about valves article on entity interpolation and what not with buffered states, but I am confused on a few things. Some source codes of interpolation I see people just store the currentTime of their own client when they receive the packet and what not, while in other source codes I see people having the server send the time the packet was sent and then using that as the time for when calculating the interpolation.
I have tried the client time way, but not the server network time way.
Is using the network time the way to get proper smoothed movement over the network?

Am I meant to also include the velocity and what not to try and lerp smoothly somehow?
For right now I am not worrying about extrapolation or anything, so is sending the velocity not needed?

Any insight is appreciated.
I have read multiple articles on this, but I guess either my expectations are too high, or I am just not sending/using enough data to properly just guess the new position instead of just relying on lerp?

[HiddenMonk]

I may have found something decent for now.
I wanted to avoid having to deal with syncing the times over the network, so instead I had the sender send a timestamp just so I could use it to help calculate the duration to be used for interpolation. I then did the whole buffering of states and what not similar to the valve article.
One thing to note is I think that although the valve articles interpolation method tries to keep you at a constant delay offset from the server movement, the code I have I am pretty sure is more so of a max delay offset, meaning its not really known as to what delay behind the senders movement you will be, but it will try to not be more than the max interpolation set.
Notice I said “try”.
The interpolation delay could go over the “expected” max interpolation since we do not take into account the total stored duration in all the states combined. We could avoid this by puting extra checks in our AddPositionState, but this might not be a problem and might even be a good thing.
This is only really an issue during large close packet loss, and should fix itself when the packet loss ends and new states come in, but overall it might even be a good thing its dynamic like this.

I have not really tested this too much, but hopefully its good.

Click for video

https://www.youtube.com/watch?v=oBSn5e-EYxk

Heres the code. Theres some debug code in there that I pushed to the left, so ignore that.
Click for code

    //-Must be used on a sequenced channel of some sort, else the timestep wrapping might cause problems.
    //-The way the interpolation works is it only starts to reach the max interpolation delay when it first starts to miss states (due to packet loss or delays).
    //Then it will keep that delay possibly always or maybe until the states get overwritten by new incoming states.
    //Could potentially reach a larger delay then expected, but thats just during lots of packet loss and can
    //also be avoided if we add extra checks in our AddPositionState to take into account all state durations combined.
    //Overall it shouldnt be a problem and should fix itself as new position states come in.
    public class SyncTransform : SyncVar
    {
        public SyncVector3 position;
        public SyncFloat rotationY;
        public SyncFloat timestamp;

        public float sendRate; //How many times a second are we sending the data?
        public float maxInterpolationCount; //How many sendRates do we want to interpolate for. Set to at least 1f or so if in no packet loss environment for smooth movement
        public float snapThresholdDistance = 2f; //How far from our targetPosition do we need to be before we decide to just teleport to it.

        Quaternion startRotation;
        Quaternion targetRotation;
        float currentRotationDuration;
        float totalRotationDuration;
        float currentPositionDuration;

        const int maxTimestamp = 64;

        //The positionStates size probably should not be any larger than maxInterpolationCount + 1,
        //else there might be great delays if theres packet loss.
        //We do +1 for a little extra buffer. Could maybe do Mathf.CeilToInt(maxInterpolationCount) as well / instead of the +1.
        int maxStoredStates {get {return (int)maxInterpolationCount + 1;}}
        float currentTimeWrapped;
        double previousTime;
        bool hasStarted;
 
        List<PositionState> positionStates = new List<PositionState>();

        Transform transform;

        public SyncTransform(Transform transform)
        {
            this.transform = transform;
            this.sendRate = SyncManager.syncClient.sendRateCount; //sendRateCount currently set to 20
            this.maxInterpolationCount = 2.25f;
        }
        public SyncTransform(Transform transform, float updateRate, float maxInterpolationCount)
        {
            this.transform = transform;
            this.sendRate = updateRate;
            this.maxInterpolationCount = maxInterpolationCount;
        }

        protected override IList<ISyncable> SetSyncables()
        {
            //The constructor parameters are (maxValue, byteSize, canBeNegative)
            position = new SyncVector3(1024, 3, true);
            rotationY = new SyncFloat(360, 2, false);
            timestamp = new SyncFloat(maxTimestamp, 2, false); //Since we use only 2 bytes, we need to wrap the value back to 0.

            //These values will be sent out every network send update, even if they didnt change.
            position.isAlwaysDirty = true;
            rotationY.isAlwaysDirty = true;
            timestamp.isAlwaysDirty = true;

            return new List<ISyncable>() {position, rotationY, timestamp};
        }

        //Only receiver should call update
        public void Update()
        {
            if(!hasStarted) return;

Vector3 prevPos = transform.position;

            currentRotationDuration += Time.deltaTime;
            currentPositionDuration += Time.deltaTime;

            HandleRotation();
            HandlePosition();

if(Mathf.Approximately(Vector3.Distance(prevPos, transform.position), 0)) { Debug.DrawRay(transform.position, Vector3.up * 3f, Color.white, 1f); }
Vector3 moveDirection = ExtVector3.Direction(prevPos, transform.position);
Color color = (ExtVector3.IsInDirection(Vector3.right, moveDirection)) ? Color.magenta : Color.yellow;
Debug.DrawRay(transform.position, (Vector3.up + moveDirection) * 2f, color, 2f);
Debug.DrawRay(transform.position, Vector3.up * (Vector3.Distance(prevPos, transform.position) / Time.deltaTime), Color.green, 2f);
        }

        void HandlePosition()
        {
            if(positionStates.Count > 1)
            {
                PositionState leftState = positionStates[0];
                PositionState rightState = positionStates[1];

                float leftOver = currentPositionDuration - rightState.duration;
                float interpolationPercent = currentPositionDuration / rightState.duration;

                transform.position = Vector3.Lerp(leftState.position, rightState.position, interpolationPercent);

                if(leftOver >= 0)
                {
                    RemoveFirstPositionState();
                    currentPositionDuration = leftOver;
                    HandlePosition();
                }
            }
            else
            {
                currentPositionDuration = 0;
            }
        }

        void HandleRotation()
        {
            transform.rotation = Quaternion.Slerp(startRotation, targetRotation, currentRotationDuration / totalRotationDuration);
        }

        //The sender calls this
        protected override void OnBeforeSerialize()
        {
            position.SetValue(transform.position);

        //We might want to handle rotation better since using eulerAngles for interpolation can cause us to rotate the long way in the wrong direction.
        //Currently we force it to always rotate in shortest path, but that isnt accurate either, but would look normal at least.
            rotationY.SetValue(transform.rotation.eulerAngles.y);

            timestamp.SetValue(SetCurrentTime());
  
            //ExtTime.realtimeSinceStartup is a self calculated time using Time.unscaledDeltaTime and stored in a double for longer lasting precision
            previousTime = ExtTime.realtimeSinceStartup;
        }

public int lossPercent = 0;
        //The receiver calls this
        protected override void OnAfterDeserialize()
        {
            hasStarted = true;

if(UnityEngine.Random.Range(0, 100) < lossPercent)
{
    Debug.DrawRay(transform.position, Vector3.down * 2f, Color.red, 1f);
    return;
}
Debug.DrawRay(transform.position, Vector3.up * 4f, Color.cyan, 2f);

            AddPositionState(new PositionState(position.value, timestamp.value));

            startRotation = transform.rotation;
            targetRotation = Quaternion.Euler(0f, rotationY.value, 0f);
            totalRotationDuration = positionStates.GetLast().duration;
            currentRotationDuration = 0;

            HandleSnapThreshold(); //Make sure to call this after we set the totalRotationDuration since this might clear the positionStates
        }

        void AddPositionState(PositionState state)
        {
            if(positionStates.Count > maxStoredStates)
            {
                RemoveFirstPositionState();
            }

            if(positionStates.Count > 0)
            {
                state.SetDuration(positionStates.GetLast().timestamp, maxTimestamp);
            }
            positionStates.Add(state);
        }

        void RemoveFirstPositionState()
        {
            positionStates.RemoveAt(0);
            currentPositionDuration = 0;
        }

        void HandleSnapThreshold()
        {
            if(positionStates.Count > 0)
            {
                Vector3 targetPosition = positionStates.GetLast().position;
                Vector3 otherPosition = (positionStates.Count > 1) ? positionStates[positionStates.Count - 1].position : transform.position;
                if(Vector3.Distance(targetPosition, otherPosition) > snapThresholdDistance)
                {
                    transform.position = targetPosition;
                    positionStates.Clear();
                    currentPositionDuration = 0;
                }
            }
        }

        float SetCurrentTime()
        {
            float elapsedTime = (float)(ExtTime.realtimeSinceStartup - previousTime);
            currentTimeWrapped += elapsedTime;

            if(currentTimeWrapped > maxTimestamp)
            {
                currentTimeWrapped = currentTimeWrapped - maxTimestamp;
                //If we are still greater than the max time, then something weird happend so just set to 0. (probably disconnected before this would happen)
                if(currentTimeWrapped > maxTimestamp)
                {
                    currentTimeWrapped = 0;
                }
            }
 
            return currentTimeWrapped;
        }

        //float GetSendRateTime(float count)
        //{
        //    return (1f / sendRate) * count;
        //}

        struct PositionState
        {
            public Vector3 position;
            public float timestamp;
            public float duration;

            public PositionState(Vector3 position, float timestamp)
            {
                this.position = position;
                this.timestamp = timestamp;
                duration = 0;
            }

            public void SetDuration(float leftStateTimestamp, float maxTimestamp)
            {
                float unwrappedTimestep = timestamp;
                if(leftStateTimestamp > timestamp)
                {
                    unwrappedTimestep = maxTimestamp + timestamp;
                }
                duration = Mathf.Max(0f, unwrappedTimestep - leftStateTimestamp);

               //I used to do it like this
               //duration = Mathf.Clamp(unwrappedTimestep - leftStateTimestamp, 0f, maxDuration); //maxDuration is calculated as GetSendRateTime(maxInterpolationCount)
               //but since we clamped to a maxDuration, it wasnt as flexible and caused jitter more often.
               //So for now we just dont clamp the duration and rely on old states being overwritten by new states to keep the interpolation delay low.
            }
        }
    }

[Vytek]

Hello @HiddenMonk ,
your info are very interesting! I am studying your code and video. Hope to check if this usefull also for my small MetaVerse in VR. It is my hobby project (GitHub - Vytek/MayaVerse: MayaVerse is an experimental Metaverse - Immersive Virtual World) using Hazel Networking Library (GitHub - Vytek/HazelTestUDPClientUnity: Unity Test Client Hazel UDP). Thank fo sharing this. Find info about, is not so easy.
Sorry for my bad englih

[Vytek]

Using Time:

Advanced Unity 3D - Syncing Multiplayer Movement in UNET (An Alternative to Network Transform)

See: GitHub - joeythelantern/UnetMovement: Syncing your multiplayer transform!