[Solved] Netcode Boss Room's Object pool retrieves from active objects.

See video below. For netcode, I decided to copy boss rooms script for object pooling and all I’m doing is spawning and firing the bullet after getting it. On collision server only, I despawn it and return it to the pool.
I have (or had, didn’t check but setting collisionchecks to true may have fixed it, it is not noticible) this issue as well with my single player custom pool and not just netcode boss room object pool version, but it is not quite as noticible in singleplayer (and host) as it is on the netcode client-side version, where it occationally gets an already active bullet, which then on server returns to shooting position and worse on client where it interpolates back to shooting position.

Notice on server-side that fired bullets are suddenly reset to starting position on fire because the object pool is retrieving active bullets as they are flying away. This is much worse looking on the client due to interpolation, where the bullet bomerangs back to starting position. (I additionally think that this will occur later on when retrieving previously returned bullets, they will spawn and interpolate to shoting point on client, but not on server.)


NcArm script getting and spawning bullet:

[ServerRpc(Delivery = RpcDelivery.Unreliable)]
    void NcArmOnFireServerRpc()
        if (ncgun == null) return;
        if (ncgun.CanFire)
            var bulletNO = NetworkObjectPool.Singleton.GetNetworkObject(NcBulletPrefab,
                ncgun.transform.position + (ncgun.transform.right * ncgun.sr.size.x / 2f), Quaternion.identity);
            var ncBullet = bulletNO.GetComponent<NcBullet>();
            ncBullet.Prefab = NcBulletPrefab;
            if (!bulletNO.IsSpawned)

NcBullet script releasing and despawning bullet on collision:

private void OnCollisionEnter2D(Collision2D collision)
        if (!IsServer) return;
        this.NetworkObject.Despawn(); // why: As Nasir suggested in the answers below,
        // returning will deactivate the object
        // Although I am unsure about wheter deactivating the object will make the code unreachable,
        // it is best to despawn before returning the object.
        // However, a flipped order did not throw error or effect the game.
        NetworkObjectPool.Singleton.ReturnNetworkObject(this.NetworkObject, Prefab);

In the NetworkObjectPool Script from Boss Room, I edited the GetNetworkObjectInternal function

    private NetworkObject GetNetworkObjectInternal(GameObject prefab, Vector3 position, Quaternion rotation)
        var queue = pooledObjects[prefab];

        NetworkObject networkObject;
        if (queue.Count > 0)
            // Here is my explaination and my custom edit just below:
            // If the network object that is picked from the queue is active,
            // and the queue still has remaining objects of the prefab type
            // (because it is greater than 0)
            // then try to get next inactive of this type.
            // And if it is still inactive after this, create a new object.

                networkObject = queue.Dequeue();
                // problem after here: It is not returned to the queue if it is active,
                 // but this is fine because eventually the bullet collides,
                // and I am able to return it. On the other hand I can
                // can re-enque it, even though it should never had been enqued,
                // and when it collides, it will enque one again when being returned?
                // set netobj to null and in the while loop check if null
            while (networkObject.isActiveAndEnabled && queue.Count > 0);
            if (networkObject.isActiveAndEnabled)
                networkObject = CreateInstance(prefab).GetComponent<NetworkObject>();
            networkObject = CreateInstance(prefab).GetComponent<NetworkObject>();

        // Here we must reverse the logic in ReturnNetworkObject.
        var go = networkObject.gameObject;

        go.transform.position = position;
        go.transform.rotation = rotation;

        return networkObject;

Yes by default it should only get objects that are not active and it should work without doing this. But somehow it picks them while they are still active and are not in queue. There is some kind of bug here?
As I was further reading, despawning destroys the object, unless it has Don’t Destroy With Owner option checked. When the object is created, I don’t think I read anywhere this option being set to true, and prehaps this is the fix.
Or what am I doing wrong? So the solution is more of a hack than a fix.