Caution: Job Scheduling Bug introduced in 2022.2.20

Unity Engine
, ,
1

In Unity 2022.2.20 a severe bug in the C# Job Scheduling System occurs causing a
SIGSEGV (Segmentation fault). As a result Unity Editor crashes while being in play mode. The bug happens sporadically, but most of the time Unity Editor crashes within the first 5 minutes being in play mode in my project. I don’t know whether the bug is ECS related or also happening when using the C# Job System standalone, but due to relying heavily on jobs it will probably more likely appear in ECS projects. The bug is also included in 2022.3.0 LTS but does NOT occur in 2022.2.19.

In case of someone from Unity is reading and interested, this is the crash log:

Unfortunately I cannot report a crash bug, because it would require uploading my project.

2

Show your ProcessShotsJob fields and where you initialise (and schedule) them.

3

I consistently see the same issue appearing in 2023.2.

4

The bug does not always happen on the same job, like ProcessShotsJob, but also on other jobs of my project. Especially on the ProcessShotsJob and the corresponding system the last relevant change I made was a couple of weeks ago and it worked flawlessly back then. That was the reason I tried some prior versions of Unity and found out that the bug was not there then.

But still here is the relevant code of the system and the job:

  [BurstCompile]
  [DisableAutoCreation]
  [WorldSystemFilter(WorldSystemFilterFlags.ServerSimulation | WorldSystemFilterFlags.ClientSimulation)]
  [UpdateInGroup(typeof(GamePredictedFixedSystemGroup))] // custom system group (only used for grouping my systems)
  [UpdateAfter(typeof(MotionSystem))]
  public partial struct ShootingProcessorSystem : ISystem
  {
    private BeginLevelSimulationCommandBufferSystem.Singleton _beginLevelSimulationCommandBufferSystemSingleton;

    private bool _isServer;
    private LoggerHandle _logger;

    private ProcessShotsJob _processShotsJob;
    private EntityQuery _processShotsQuery;
    private FixedString32Bytes _worldName;

    /// <inheritdoc />
    public void OnCreate(ref SystemState state)
    {
      state.DetermineShortWorldName(out this._worldName);
      this._logger = LogFactory.CreateLogger(nameof(ShootingProcessorSystem));
      Log.To(this._logger).Info("[{SystemTime}/{WorldName}] Creating system", SystemAPI.Time.ElapsedTime, this._worldName);

      // System data
      this._isServer = state.WorldUnmanaged.IsServer();

      // Jobs
      this._processShotsJob = new ProcessShotsJob()
      {
        // Various
        JobLogger = LogFactory.CreateLogger(nameof(ShootingProcessorSystem), nameof(ProcessShotsJob)),
        WorldName = this._worldName,

        // Buffers and arrays
        RaycastHits = new NativeList<RaycastHit>(20, Allocator.Persistent),
      };
      this._processShotsJob.Init(ref state);

      // Queries
      this._processShotsQuery = SystemAPI.QueryBuilder()
        .WithAspect<ShootingAspect>()
        .Build();

      // Get singleton of custom command buffer system
      // (Buffer is implemented as described here: https://docs.unity3d.com/Packages/com.unity.entities@1.0/manual/systems-entity-command-buffer-automatic-playback.html)
      this._beginLevelSimulationCommandBufferSystemSingleton = SystemAPI.GetSingleton<BeginLevelSimulationCommandBufferSystem.Singleton>();

      // Requires
      state.RequireForUpdate<PhysicsWorldSingleton>();
      state.RequireForUpdate<PhysicsWorldHistorySingleton>();
      state.RequireForUpdate<NetworkTime>();
      state.RequireForUpdate(this._processShotsQuery);
    }

    /// <inheritdoc />
    [BurstCompile]
    public void OnDestroy(ref SystemState state)
    {
      this._processShotsJob.RaycastHits.Dispose();
    }

    /// <inheritdoc />
    [BurstCompile]
    public void OnUpdate(ref SystemState state)
    {
      if (!NetcodeUtilities.TryGetNetworkTime(SystemAPI.GetSingleton<NetworkTime>(), out var networkTime))
      {
        return;
      }

      this._processShotsJob.CommandBuffer = this._beginLevelSimulationCommandBufferSystemSingleton.CreateCommandBuffer(state.WorldUnmanaged);
      this._processShotsJob.NetworkTime = networkTime;
      this._processShotsJob.PhysicsWorldSingleton = SystemAPI.GetSingleton<PhysicsWorldSingleton>();
      this._processShotsJob.PhysicsWorldHistorySingleton = SystemAPI.GetSingleton<PhysicsWorldHistorySingleton>();

      ref var job = ref this._processShotsJob;
      job.Update(ref state);
      state.Dependency = job.ScheduleByRef(this._processShotsQuery, state.Dependency);
    }   

    [BurstCompile]
    private partial struct ProcessShotsJob : IJobEntity
    {
      public EntityCommandBuffer CommandBuffer;

      [ReadOnly]
      public ComponentLookup<CommandDataInterpolationDelay> CommandDataInterpolationDelayLookup;

      [ReadOnly]
      public bool IsServer;

      [ReadOnly]
      public LoggerHandle JobLogger;

      [ReadOnly]
      public ComponentLookup<LinkedConnection> LinkedConnectionLookup;

      [ReadOnly]
      public ComponentLookup<LinkedInputNode> LinkedInputNodeLookup;

      [ReadOnly]
      public ComponentLookup<NetworkId> NetworkIdLookup;

      [ReadOnly]
      public NetworkTime NetworkTime;

      [ReadOnly]
      public PhysicsWorldHistorySingleton PhysicsWorldHistorySingleton;

      [ReadOnly]
      public PhysicsWorldSingleton PhysicsWorldSingleton;

      [ReadOnly]
      public BufferLookup<PlayerInputElement> PlayerInputElementLookup;

      [ReadOnly]
      public BufferLookup<UnitAnimationHistoryStateElement> UnitAnimationHistoryStateElementLookup;

      [ReadOnly]
      public Entity PureImmediateShotPrefabEntity;

      public NativeComparers.RaycastHitDistanceComparer RaycastHitDistanceComparer;
      public NativeList<RaycastHit> RaycastHits;

      [ReadOnly]
      public WeaponDefinitionPoolReference WeaponDefinitionPoolReference;

      [ReadOnly]
      public FixedString32Bytes WorldName;

      public void Init(ref SystemState state)
      {
        var prefabContainer = state.GetEntityQuery(ComponentType.ReadOnly<PurePrefabContainer>()).GetSingleton<PurePrefabContainer>();
        this.PureImmediateShotPrefabEntity = prefabContainer.PureImmediateShotPrefab;

        this.IsServer = state.WorldUnmanaged.IsServer();

        this.LinkedConnectionLookup = state.GetComponentLookup<LinkedConnection>(true);
        this.NetworkIdLookup = state.GetComponentLookup<NetworkId>(true);
        this.CommandDataInterpolationDelayLookup = state.GetComponentLookup<CommandDataInterpolationDelay>(true);
        this.LinkedInputNodeLookup = state.GetComponentLookup<LinkedInputNode>(true);
        this.PlayerInputElementLookup = state.GetBufferLookup<PlayerInputElement>(true);
        this.UnitAnimationHistoryStateElementLookup = state.GetBufferLookup<UnitAnimationHistoryStateElement>(true);

        this.WeaponDefinitionPoolReference = state.GetEntityQuery(ComponentType.ReadOnly<WeaponDefinitionPoolReference>()).GetSingleton<WeaponDefinitionPoolReference>();
      }

      [BurstCompile]
      public void Update(ref SystemState state)
      {
        this.LinkedConnectionLookup.Update(ref state);
        this.NetworkIdLookup.Update(ref state);
        this.CommandDataInterpolationDelayLookup.Update(ref state);
        this.LinkedInputNodeLookup.Update(ref state);
        this.PlayerInputElementLookup.Update(ref state);
        this.UnitAnimationHistoryStateElementLookup.Update(ref state);
      }

      [BurstCompile]
      public void Execute(ShootingAspect shootingAspect)
      {
        // ...
      }
    }

    private readonly partial struct ShootingAspect : IAspect
    {
      public readonly RefRO<LinkedFaction> LinkedFactionRef;

      [Optional]
      public readonly RefRO<LinkedPlayer> LinkedPlayerRef;

      public readonly RefRO<LocalTransform> LocalTransformRef;
      public readonly RefRO<PureUnit> PureUnitRef;

      public readonly RefRW<UnitAttacking> UnitAttackingRef;
      public readonly Entity UnitEntity;
      public readonly RefRO<UnitMotion> UnitMotionRef;
      public readonly RefRW<UnitWieldedWeapon> UnitWieldedWeaponRef;
    }
  }

  [BurstCompile]
  public static class NativeComparers
  {
    [BurstCompile]
    public struct RaycastHitDistanceComparer : IComparer<RaycastHit>
    {
      public float3 Origin;

      [BurstCompile]
      public int Compare(RaycastHit a, RaycastHit b)
      {
        var aDistance = math.distance(this.Origin, a.Position);
        var bDistance = math.distance(this.Origin, b.Position);

        if (aDistance < bDistance)
        {
          return -1;
        }
        else if (aDistance > bDistance)
        {
          return 1;
        }
        else
        {
          return 0;
        }
      }
    }
  }

Interessting, thanks for sharing this info. So it is not a specific project releated issue in your or my project.

5

In my case it’s the same job each time

6

You can actually submit a bugreport (crashdump). In the crash reporter, on the bottom, click the project and then remove. This will upload the crashdump only which is usually enough for crashes in native code.

7

Maybe one of your systems is running into an unguarded safety violation?

8

I have two different jobs causing the crash. One is an IJobEntity (stated in my post above) and one is an IJob. Interesstingly the error occurs for an IJobParallelFor in your case. In my case most of the time the crash is caused by scheduling the IJob so it seems to be more likely happening when using jobs which are NOT tied to IComponentData and the ECS workflow.

Thanks for sharing the info. In the stated case I use an aspect as Execute parameter which is also included in the entity query, so all should be fine here. I do not use InternalCompilerInterface anywhere in my project.

Thanks for the info. I was not aware of this. I created and sent the crash report to Unity including the dumps but excluding the project itself. If I got news on this I will post here.

9

Updated to 2023.2.18 and now it is much worse than before. It now takes less than a minute for editor to crash (total seconds in play mode summarized).
Downgraded back to 2023.2.17 and can run for minutes without a crash.
So it does look like a leak/regression in the core.

10

My crash report was categorized as duplicate. Unity already knows about this bug. This is the issue tracker link:

11

Happening to me as well after upgrading from 2022.2.18 to 2022.3.1 and from 1.0.0pre65 to 1.0.10.

Commenting out the specific job only causes a different job to crash on the JobsUtility:Schedule_Injected function, and it consistently happens after roughly 20 seconds of running in playmode.

12

It is very inconsistent but I am also seeing this. It is a schedule-time crash, meaning it is crashing on the main thread and has nothing to do with the user Execute method inside the job.

13

Seeing this very rarely in 2022.3.0f1 (after long periods in play mode), but seeing it every time ~20 seconds after entering play mode in 2022.3.1f1.

14

[BurstCompile]OnUpdate(ref SystemState state){

var job1 = new Job1();
state.Dependency = job1.Schedual(state.Dependency);

var job2 = new Job2();
state.Dependency = job2.SchedualParallel(state.Dependency);
}

Crash on JobsUtility:Schedule_Injected.

Removing [BurstCompile] solved this problem.

15

I have stayed with Version 2022.2.19, but because I hade some other issues I tried whether or not they also appear in 2022.3.4. But I noticed the crash caused by the scheduling bug does no more occur in my project (without and with Burst compilation). Played three times ten minutes in a row without Editor crash. But the corresponding issue is still pending. The [BurstCompile] atttribute is still their in the system and job, which originally caused the crash since 2022.2.20 in my project. Can someone approve or disapprove that the crash bug has been fixed in his project in Unity 2022.3.4?