NativeArray.CopyTo to larger managed array

Unity Engine
,
1

Hi.

When using the CopyTo function to copy data from a NativeSlice or a NativeArray to a managed array there is internal validation to make sure the managed array has the same size.

Would it be possible to allow copying if the managed array is larger than the native array.
I have a use case where I have a pre allocated managed array that in some cases are larger than the nativearray produced by jobs and I do not want to allocate this again and produce GC

I do this now with a custom CopyTo function and MemCopy but I would like to get the unsafe code out of the project.

Lennart

2 Likes
2

You can use NativeSlice and Slice function to get a subsection of the array.

2 Likes
3

Yes. I use slices. But if I have a slice of 900 items and want to copy this to a pre allocated array of 1023 (for instanced render api) it will not do that since the size does not match. I would like the CopyTo to just write to the first 900 items in the array.

4

Can you just use list.AddRange(nativeArray)?

5

That would be much slower than a single memory copy.

What I would like is basically this. Where it allows for copying the slice to a array that is larger than the slice. The CopyTo needs the size to be the same.

This code works good but requires the unsafe code checkbox to be enabled. Would be nice if the built-in copyTo could do the same.

  public static unsafe void CopyToFast<T>(
            this NativeSlice<T> nativeSlice,
            T[] array)
            where T : struct
        {         
            if (array == null)
            {
                throw new NullReferenceException(nameof(array) + " is null");
            }           
            int nativeArrayLength = nativeSlice.Length;
            if (array.Length < nativeArrayLength)
            {
                throw new IndexOutOfRangeException(
                    nameof(array) + " is shorter than " + nameof(nativeSlice));
            }
            int byteLength = nativeSlice.Length * UnsafeUtility.SizeOf<T>();
            void* managedBuffer = UnsafeUtility.AddressOf(ref array[0]);
            void* nativeBuffer = nativeSlice.GetUnsafePtr();
            UnsafeUtility.MemCpy(managedBuffer, nativeBuffer, byteLength);
        }
2 Likes
6

The current NativeArray.CopyTo(T[ ]) is not a single memory copy, it copies each element to the destination individually. Even if they adjusted the method to allow copying to an array of a larger size, your method would still be faster (in most cases) since it is a single memory copy. That said, it would definitely be nice to see something like this in the Unity library.

7

Joachim Ante confirmed that in 2018.3 all the copy functions use the memory copy internally.

3 Likes
8

I would also be interested in a method to copy the content of a NativeArray to part of a managed List. So far I came up with this that crashes unity…

    static Dictionary<Type, FieldInfo> fieldCache = new Dictionary<Type, FieldInfo>();
    public unsafe static void Copy<T>(this List<T> list, int listStart, NativeArray<T> array, int arrayStart, int count) where T : struct
    {
        void* source = NativeArrayUnsafeUtility.GetUnsafePtr(array);

        FieldInfo field;
        if (!fieldCache.TryGetValue(typeof(List<T>), out field))
        {
            field = typeof(List<T>).GetField("_items", BindingFlags.Instance | BindingFlags.NonPublic);
            fieldCache.Add(typeof(List<T>), field);
        }

        object listItems = field.GetValue(list);

        ulong gcHandle;
        void* destination = UnsafeUtility.PinGCObjectAndGetAddress(listItems, out gcHandle);

        UnsafeUtility.MemCpy(destination, source, count * UnsafeUtility.SizeOf(typeof(T)));

        UnsafeUtility.ReleaseGCObject(gcHandle);
    }

EDIT:

void* destination = UnsafeUtility.PinGCObjectAndGetAddress(listItems, out gcHandle);

This is probably wrong. I’m probably getting the address of the array and not of its data.

9

Use my function from the post above and change the index of the managed stay you get the pointer from.

10

Ah, thanks for making me re-read it a third time. I finally got the bit I was missing :slight_smile:
That ref array[0], so simple…

11

You can do a single copy like that to multi dimension arrays also

12

Just hit upon this problem. Another unity3d YAGNI.
I DEMAND :wink: functionality that let me copy NativeArray data to another NativeArray of different size.
It is just a common sense.

13

@koirat sounds like Unity - Scripting API: Unity.Collections.NativeArray_1.Copy to me

5 Likes
14

thx. looks like it is a static method.

15

Alternatively you can use slices, eg:
target.Slice([start], [length]).CopyFrom(source.Slice([start], [length]))

2 Likes
16

In that case using GetSubArray would probably be faster unless you need it to handle stride.

1 Like
17

I’m pretty sure MemCpyStride simply forwards to MemCpy when sourceStride==destStride==elementSize but better be safe.

18

The reason GetSubArray is better is because NativeArray is faster than NativeSlice since the compiler can assume there is not a dynamic offset between elements. NativeSlice should only be used if it is actually necessary to have dynamic strides.

1 Like