-edit-
I’ve renamed the title to better reflect the different things in this thread now.
IJobProcessNativeMultiHashMap<TKey, TValue> is available here: Iterating NativeMultiHashMap - Unity Engine - Unity Discussions
NativeMultiHashMap.GetEnumerator() is available here: Iterating NativeMultiHashMap - Unity Engine - Unity Discussions
-original-
It’s been an often requested feature, and I’ve come into the need on occasion to be able to iterate a NativeMutliHashMap in a job. At the moment, all we have is IJobNativeMultiHashMapMergedSharedKeyIndices which only works on NativeMutliHashMap<int, int> and does not pass key value.
I could not be bothered waiting any longer so I decided to write a new job to do this myself. The interface looks like so,
/// <summary>
/// Iterates a NativeMultiHashMap.
/// </summary>
/// <typeparam name="TKey">The key.</typeparam>
/// <typeparam name="TValue">The value.</typeparam>
public interface IJobProcessNativeMultiHashMap<in TKey, in TValue>
where TKey : struct, IEquatable<TKey>
where TValue : struct
{
/// <summary>
/// Called for every key, value pair of the <see cref="NativeMultiHashMap{TKey,TValue}"/>
/// </summary>
/// <param name="key">The value of the key.</param>
/// <param name="value">The value of the pair.</param>
void Execute(TKey key, TValue value);
}
The schedule looks like so,
public static unsafe JobHandle Schedule<TJob, TKey, TValue>(this TJob jobData, NativeMultiHashMap<TKey, TValue> hashMap, int minIndicesPerJobCount, JobHandle dependsOn = default)
where TJob : struct, IJobProcessNativeMultiHashMap<TKey, TValue>
where TKey : struct, IEquatable<TKey>
where TValue : struct
Pretty much the same as IJobNativeMultiHashMapMergedSharedKeyIndices except takes any NativeMultiHashMap.
Here is a quickly thrown together unit test so you can see how you’d implement / use it.
[Test]
public void JobProcessNativeMultiHashMap()
{
const int keyCount = 10;
const int valueCount = 10;
var values = new NativeMultiHashMap<double, double>(keyCount * valueCount, Allocator.TempJob);
var random = new Random(1234);
for (var i = 0; i < keyCount; i++)
{
var key = random.NextDouble();
for (var j = 0; j < valueCount; j++)
{
var value = random.NextDouble();
values.Add(key, value);
}
}
var results = new NativeMultiHashMap<double, double>(keyCount * valueCount, Allocator.TempJob);
var job = new JobTest
{
Results = results.ToConcurrent(),
};
var handle = job.Schedule(values, 1);
handle.Complete();
Assert.AreEqual(values.Length, results.Length);
values.Clear();
results.Clear();
}
[BurstCompile]
private struct JobTest : IJobProcessNativeMultiHashMap<double, double>
{
public NativeMultiHashMap<double, double>.Concurrent Results;
/// <inheritdoc />
public void Execute(double key, double value)
{
this.Results.Add(key, value);
}
}
At this stage I’m not 100% sure how the threading on this works as I’m still not certain of the workings of NativeMultiHashMap and I can’t tell if each key will operate on it’s own thread (what i’m hoping for) or they will mix across threads. I need to do more testing and reading the source.
Now I’m not releasing it right this second, maybe later today/tomorrow if I can figure out how the work is split but I also wanted feedback on the interface. If you were to use this, is void Execute(TKey key, TValue value) what you’d want or is there an alternative? I can’t really think of one with how the NativeMultiHashMap is setup.
-side note-
The biggest challenge of getting this to work was the fact that everything is internal for the NativeMutliHashMap. My solution if anyone was interested was taking advantage of the sequential memory layout so I wrote a couple of imposters
[StructLayout(LayoutKind.Sequential)]
public unsafe struct NativeMultiHashMapImposter<TKey, TValue>
where TKey : struct, IEquatable<TKey>
where TValue : struct
{
[NativeDisableUnsafePtrRestriction] internal NativeHashMapDataImposter* m_Buffer;
#if ENABLE_UNITY_COLLECTIONS_CHECKS
AtomicSafetyHandle m_Safety;
[NativeSetClassTypeToNullOnSchedule] DisposeSentinel m_DisposeSentinel;
#endif
Allocator m_AllocatorLabel;
}
[StructLayout(LayoutKind.Sequential)]
public unsafe struct NativeHashMapDataImposter
{
public byte* values;
public byte* keys;
public byte* next;
public byte* buckets;
public int capacity;
public int bucketCapacityMask; // = bucket capacity - 1
// Add padding between fields to ensure they are on separate cache-lines
private fixed byte padding1[60];
public fixed int firstFreeTLS[JobsUtility.MaxJobThreadCount * IntsPerCacheLine];
public int allocatedIndexLength;
// 64 is the cache line size on x86, arm usually has 32 - so it is possible to save some memory there
public const int IntsPerCacheLine = JobsUtility.CacheLineSize / sizeof(int);
}
and remapped the hash map to them
public static implicit operator NativeMultiHashMapImposter<TKey, TValue>(NativeMultiHashMap<TKey, TValue> hashMap)
{
var ptr = UnsafeUtility.AddressOf(ref hashMap);
UnsafeUtility.CopyPtrToStructure(ptr, out NativeMultiHashMapImposter<TKey, TValue> imposter);
return imposter;
}
It seems to work fantastic, but if anyone can tell me why this is a very bad idea it would be great to know now!