How Can I convert a byte array to floats using NativeArrays and Unity's Job System?

Unity Engine
1

I am trying to multithread the conversion of bytes to floats using unity’s new job system. Previously, I was using System.BitConverter.ToSingle on the main thread and that was working perfectly. However, since arrays are not supported in the job system, I must use a NativeArray instead. And System.BitConverter.ToSingle does not accept a NativeArray. Converting the NativeArray to a normal array works but slows things down way too much since accessing managed memory from within a job is very slow. So instead, I looked at the source code for BitConverter and tried to just copy what they were doing which seemed simple enough. So I have:

using UnityEngine;
using Unity.Collections;
using Unity.Jobs;
using Unity.Burst;


[BurstCompile]
struct GetVerticesJob : IJobParallelFor
{
   [ReadOnly]
   public NativeArray<byte> bufferData;

   public NativeArray<Vector3> vertices;

   public int offset;

   public void Execute(int i)
   {
       int step = 4;
       int localOffset = offset + (12 * i);
       vertices[i] = new Vector3(
           ConvertBytesToFloat(localOffset + (step * 0)),
           ConvertBytesToFloat(localOffset + (step * 1)),
           ConvertBytesToFloat(localOffset + (step * 2))
       );
   }

   float ConvertBytesToFloat(int offset)
   {
       int value = bufferData[offset]  | bufferData[offset + 1] << 8 | bufferData[offset + 2] << 16 | bufferData[offset + 3] << 24;
       return (float)value;
   }

}

However, this results in very wrong and enormous values for the floats. What am I doing wrong and how can I get this working? Thanks!

2

I think your order is wrong

bufferData[offset] << 24 | bufferData[offset + 1] << 16 | bufferData[offset + 2] << 8 | bufferData[offset + 3];

That said, I don’t think your (float) int conversion is going to work either.
Because you’re just converting an int directly into a float. It’s never going to have decimal places.
I doubt this is the expected result you want.

3

Thanks for the response!

BitConverter does:

                 if( IsLittleEndian) {
                        return (*pbyte) | (*(pbyte + 1) << 8)  | (*(pbyte + 2) << 16) | (*(pbyte + 3) << 24);
                    }
                    else {
                        return (*pbyte << 24) | (*(pbyte + 1) << 16)  | (*(pbyte + 2) << 8) | (*(pbyte + 3));                      
                    }

I tried for both littleEndian and bigEndian (your solution) and both give me the wrong values. But at least with littleEndian the sign is always correct.

And yes, I think you’re correct about the cast not working. Not sure what else to try though. Any ideas?

4

Anyway if you don’t need a copy of your data you can probably just reinterpret the array.

public NativeArray<byte> bufferData;

NativeArray<float> floats = bufferData.Reinterpret<float>(UnsafeUtility.SizeOf<float>());

// these point to same memory, modifying one array will modify the other

I think this will only work if it’s big-endian (haven’t tested this at all).

1 Like
5

Nop, this will work for both (considering both data and platform are using the same). If the source is little endian too, your example with Reinterpret is the best solution IMHO. But using Reinterpret could be done without the job.

About the endianness support in Unity: https://discussions.unity.com/t/721887/13

[ ]'s

6

Doesn’t look like Reinterpret is defined on NativeArray. And don’t see it in the documentation either.

7

2019.3+

8

IIRC it’s only in 2019.3 or later.

[ ]'s

9

Ah, ok. Thanks so much to you both! Will update unity, try again, and report back.

10

I also think unity mathematics has float ↔ int functions.

11

Update unity. Now when I do:

nativeBufferBytes = new NativeArray<byte>(bufferData, Allocator.Persistent);
nativeBufferFloats = nativeBufferBytes.Reinterpret<float>(UnsafeUtility.SizeOf<float>());

I get ‘InvalidOperationException: Type System.Byte was expected to be 4 but is 1 bytes’

12

Sorry this should be

nativeBufferBytes.Reinterpret(UnsafeUtility.SizeOf())

just did a quick test to check converting forward and back

            var floats = new NativeArray<float>(1, Allocator.Persistent);
            floats[0] = 13.145f;

            var bytes = floats.Reinterpret<byte>(UnsafeUtility.SizeOf<float>());
            Debug.Log(bytes.Length);

            var floats2 = bytes.Reinterpret<float>(UnsafeUtility.SizeOf<byte>());
            Debug.Log(floats2.Length);
            Debug.Log(floats2[0]);

            floats.Dispose();

result

4
1
13.145

as expected

3 Likes
13

Ah, perfect. It works now. Thank you!!
This is without using jobs but I guess jobs aren’t actually needed here.
Do you know what the performance implications of Reinterpret are?
the documentation says this ‘creates a view into memory’. Does that mean the new NativeArray does not actually use any memory?

14

there is close to no performance cost.

both arrays point to the same memory. making a change on 1 array will modify the other.
if you don’t want this behaviour just make a copy of the array after reinterpretting.

15

I don’t need to make any changes to the array at all so don’t need to copy. This is perfect.
Is it possible to reinterpret the bytes as an array of vector3s of floats?

16

you can reinterpret to anything (that can be used in a native array), even if it doesn’t make sense

it’s not magical. the memory layout has to match.

17

yeah, wow.
just did this and it worked

NativeArray<Vector3> vertices = nativeBufferBytes.GetSubArray(offset, bufferView.byteLength).Reinterpret<Vector3>(1);

Inspected in the profiler and yeah, it’s instant. That’s crazy.
Converting back to a regular array takes some time though, of course.

Thank you so much for all your help!!

18

i have error
Library\PackageCache\com.unity.2d.common@2.0.2\Runtime\InternalBridge\InternalEngineBridge.cs(17,48): error CS1503: Argument 2: cannot convert from ‘Unity.Collections.NativeArray’ to ‘Unity.Collections.NativeArray<UnityEngine.Vector3>’
while updating package manager.
Any help?

19

How do I go about getting a string out of a job?

I’ve created a job with a NativeString4096 field, i’m passing a ref to one from outside the job, i’m setting the value inside the job, but after the job Completes the result is still empty.

Shouldn’t it write to it if i pass it by ref to the constructor of the job?

NativeString4096 results = new NativeString4096();
MyJob job = new MyJob(ref results);
JobHandle jobHandle = job.Schedule();
jobHandle.Complete();
Debug.Log(results);

public struct StringJob : IJob
    {
        public NativeString4096 result;

        public StringJob(ref NativeString4096 nativeString)
        {
            result = nativeString;
        }
       
        public void Execute()
        {
            result = "Hello";
        }
    }
20

NativeString4096 is a value-type. If you set it in the job, it won’t be copied back to the original struct that you used to schedule the job. You have to allocate the string somewhere that’s independent of the job, e.g. in a NativeArray<NativeString4096> of length 1:

struct StringJob : IJob {
    public NativeArray<NativeString4096> result;
    public void Execute() {
        result[0] = new NativeString4096("Hello");
    }
}