How to convert UnsafeList to array[] and vice versa?

[Taigama]

• I am working with parallel job.
• The target is decoding a lot of binary data files with the power of multithread.
• Assume each file is a byte[] with around 7~10 Megabytes after decoded.
• The problem is I don’t know the exact size of decoded data until it gets decoded, so I have to use UnsafeList instead of NativeArray for the output.
• But UnsafeList doesn’t provide any direct conversion to/from array[], then I loop among huge iteration to append each byte of the data. It slows my device alot, are there any proper way?

Here is the sample codes:

• data structure.

public struct InputData
{
    [ReadOnly] public NativeArray<byte> encoded;
}
public struct OutputData
{
    [WriteOnly] public int index;

    // I want to use NativeArray instead, but the Job doesn't allow Allocator.Persistent for NativeArray.
    // I need the data allocated in Persistent to be process later in the main thread
    [WriteOnly] public UnsafeList<byte> decoded;
}

• job structure. Assume that the job is scheduled and works properly.

[BurstCompile]
public struct DecodeDataJob : IJobParallelFor
{
    // this nested NativeArray inside UnsafeList
    // UnsafeList<InputData> has been allocated when schedule job
    [ReadOnly] public UnsafeList<InputData> inputs;

    // this nested UnsafeList inside UnsafeList
    // UnsafeList<OutputData> has been allocated when schedule job
    [WriteOnly] public UnsafeList<OutputData>.ParallelWriter outputWriter;
    
    public void Execute(int index)
    {
        InputData anInput = inputs[index];
        
        // this shows warning: NativeArray ToArray() contain managed types
        byte[] arrayInput = anInput.encoded.ToArray();

        // this also shows warning: contain managed types
        byte[] arrayOutput = DecodeData(arrayInput);
        
        // - Are there better way to convert byte[] to UnsafeList<byte>? -
        UnsafeList<byte> output = new UnsafeList<byte>(arrayOutput.Length, Allocator.Persistent);
        for (int i = 0; i < arrayOutput.Length; ++i)
        {
            output.AddNoResize(arrayOutput[i]);
        }

        OutputData anOutput = new OutputData()
        {
            index = index,
            decoded = output
        };
        outputWriter.AddNoResize(anOutput);
    }

    // this function is from a library, which I can't change the input and output 
    private static byte[] DecodeData(byte[] input) { }
}

• managed class for post process.

public class ProcessDecodedDataInManaged
{
    public void ParseDecodedFiles(in UnsafeList<OutputData> outputs)
    {
        for (int i = 0; i < outputs.Length; ++i)
        {
            OutputData anOutput = outputs[i];
            byte[] arrayOutput = this.ConvertUnsafeListToArray(anOutput.decoded);
            anOutput.decoded.Dispose();
            ProcessDecoded(arrayOutput);
        }
    }
    
    private byte[] ConvertUnsafeListToArray(in UnsafeList<byte> input)
    {
        byte[] result = new byte[input.Length];
        for (int i = 0; i < result.Length; ++i)
        {
            result[i] = input[i];
        }
        return result;
    }

    // this function is from a library, which I can't change the input and output 
    private static void ProcessDecoded(byte[] decoded) { }
}

The ConvertUnsafeListToArray causes a huge drop in performance with just 7 files.

For 42 files, it hanged.

[Spy-Master]

Advice for the route you’re going down right now

As a general pattern, I would recommend using NativeArray containing elements with unsafe collections, i.e. NativeArray for DecodeDataJob.inputs/DecodeDataJob.outputWriter (the given code always adds one item per iteration and suggests you should just write directly to the array element for the given index instead of adding to a list) and UnsafeList for InputData.encoded and OutputData.decoded to avoid Unity arguing about nested native containers.

You can’t introduce managed arrays anywhere in Burst code. Work involving managed arrays always needs to be done outside Burst. Ideally, you would instead call into Burst-compatible plugin code that takes simple Span/ReadOnlySpan, NativeArray, or even just pointer + length.

Array conversion can be simplified to a simple memcpy which is the standard way to copy data:

public static unsafe T[] ToArray<T>(this in UnsafeList<T> list) where T : unmanaged
{
    T[] result = new T[list.Length];
    fixed (T* resultPtr = result)
    {
        UnsafeUtility.MemCpy(resultPtr, list.Ptr, (long)list.Length * sizeof(T));
    }
    return result;
}

public static void AppendFromArray<T>(ref UnsafeList<T> list, T[] array) where T : unmanaged
{
    int offset = list.Length;
    list.Resize(offset + array.Length, NativeArrayOptions.UninitializedMemory);
    fixed (T* arrayPtr = array)
    {
        UnsafeUtility.MemCpy(list.Ptr + offset, arrayPtr, (long)array.Length * sizeof(T));
    }
}

Advice for how else you can do this

It seems quite ridiculous to jump through hoops of converting between arrays and unity collection lists. In my view, it would be vastly more efficient and cleaner to instead pass managed objects via GCHandle, like a managed array where each element stores both the input and output arrays, and ignore unity collections altogether.

class MyIOElement
{
    public byte[] Input;
    public byte[] Output;
}

struct Job1 : IJobParallelFor
{
    public GCHandle ArrayHandle;

    public void Execute(int index)
    {
        MyIOElement element = (ArrayHandle.Target as MyIOElement[])[index];
	element.Output = DoThing(element.Input);
    }
}

MyIOElement[] array = ...;
GCHandle handle = GCHandle.Alloc(array);
var jobHandle = new Job1 { ArrayHandle = handle }.Schedule();
// must complete job before freeing handle
jobHandle.Complete();
handle.Free();

[Taigama]

Thank you. I didn’t know that I can write to NativeArray directly in parallel without the need of using a parallel writer.

By the way, can you correct the code a bit (for future reader)?

• For function ToArray() the MemCpy() require input the size in long instead of ulong, also in AppendFromArray() function.
• For function AppendFromArray() line 15, it should be fixed (T* arrayPtr = arrayPtr), replace arrayPtr with array.