How to pass the intermediate data in NativeQueue from one parallel job to another?

Unity Engine
,
1

Hi, I’ve a system that has two parallel jobs,

  • the first one will run through chunks and populate some intermediate data into a NativeQueue,
  • the second job is expected to work on the pass-in NativeQueue and get the final result.

The problem is that the second job (IJobParallelFor) needs the length to schedule, which is not available at the point in main thread. I checked the IJobParallelForDefer but it needs a NativeList, which doesn’t have a Concurrent version to replace the NativeQueue in the first job.

It seems that the only available method is to use a NativeHashMap? Or is there something I’m missing here?

2

I don’t get your second job.

How are you processing a NativeQueue in an IJobParallelFor

3

Oops… you’re right. It’s not feasible from the start.

So the only available method is via NativeHashMap/NativeMultiHashMap? But we have to know a maximum capacity to use the concurrent hashmap. Hmm…

4

This is a common issue with no single solution. It depends a lot on your data, I have 2 ways that I often go about it, both of which you mentioned, but there are alternatives.

The first way is just an intermezzo job that copies the queue to a list. Common enough that I threw together a generic version of this some point in the past.

// <copyright file="CopyQueueToListJob.cs" company="BovineLabs">
// Copyright (c) BovineLabs. All rights reserved.
// </copyright>

namespace BovineLabs.Common.Jobs
{
    using Unity.Burst;
    using Unity.Collections;
    using Unity.Jobs;

    /// <summary>
    /// Copy a <see cref="NativeQueue{T}"/> to a <see cref="NativeList{T}"/>.
    /// </summary>
    /// <typeparam name="T">The type of the containers.</typeparam>
    [BurstCompile]
    public struct CopyQueueToListJob<T> : IJob
        where T : struct
    {
        /// <summary>
        /// The input queue.
        /// </summary>
        public NativeQueue<T> Queue;

        /// <summary>
        /// The output list.
        /// </summary>
        public NativeList<T> List;

        /// <inheritdoc/>
        public void Execute()
        {
            this.List.Clear();
            this.List.Capacity = this.Queue.Count;

            while (this.Queue.TryDequeue(out var v))
            {
                this.List.Add(v);
            }
        }
    }
}

It surprisingly takes little time to do this in a burst job even with a lot of elements. However if length is an issue, the other method is to use a hash map if you can figure out an upper limit of its capacity. This is usually from Query.CalculateLength() if each entity in a query will at most add 1 to the map.

        {
            // Ensure our hash map have capacity
            var requiredCapacity = this.allObserversQuery.CalculateLength();
            if (requiredCapacity > this.visionMap.Capacity)
            {
                this.visionMap.Capacity = requiredCapacity;
            }

            // .. schedule job
1 Like