How do I determine the batch size to assign to a job to take advantage of parallel processing

Unity Engine
, , , , , ,
1

I’ve always been perplexed by the idea behind batch sizes when scheduling a parallel job.

My question essentially is: How do I know the batch size to put into the Schedule method?

The documentation suggests that A simple job, for example adding a couple of Vector3 to each other should probably have a batch size of 32 to 128. However if the work performed is very expensive then it is best to use a small batch size, for expensive work a batch size of 1 is totally fine.

How do I know if the work is expensive enough to need a batch size of 1? And does it mean that while working on a simple task, any size specifically between 32 to 128 is ok? What happens if I set the batch size to be any other value…say 2, 3, 20, 1000, 5000, etc?

2

There’s a small overhead for fetching a new batch when one completes. Using a batch size of 32 means this overhead happens 32 times less often. However, larger batches decrease the ability for a thread that finishes early to find other work. You still want all your threads to finish a parallel job at the same time. That’s the balancing act. Rarely does a single power of two in batch size make a significant difference in performance. It has to cascade.

3

Found this logic in the svelto codebase which I think chooses a pretty reasonable default:


    public static int ChooseBatchSize(int totalIterations)
    {
        var iterationsPerBatch = totalIterations / Environment.ProcessorCount;

        if (iterationsPerBatch < 64)
        {
            return 64;
        }

        return iterationsPerBatch;
    }
4

I just return the max between 1 and totalIterations / Environment.ProcessorCount

5

Both of these above two posts will defeat the benefits of work-stealing. If one thread starts late or runs slow, the other threads won’t be able to help.

6

You need to profile. The performance of the job determines the batch size.

In general, if a job completes in 10us or less, I will run it single-threaded. If I parallelize a job that completes in less than 10us, I often find that it takes about the same amount of time, but across more CPUs, thereby wasting time on other CPUs.

If a job takes longer than this, I experiment with different batch sizes until the job hits peak performance. There usually comes a point where increasing the parallelism can make the job complete e.g. 20% faster, but it may take e.g. 50% more CPU time in total across all CPUs. In these cases the correct solution varies.

I use a 7950X3D, and I run the Unity Editor on my cache cores. This means I end up optimizing job scheduling (and my game in general) for Ryzen 7000 CPUs with 3D V-Cache. It’s what I have, so that’s just how it goes.

My CPU is pretty fast, so a 10us job is very likely to be slower on other CPUs. This means that other CPUs would probably benefit from smaller batch sizes. I’m not sure how we are expected to account for that.