"Fatal error in gc: Too many heap sections"

Unity Engine
1

Since 2018.3, we’ve been getting this error frequently while doing an allocation-intensive operation:

Admittedly, we allocate a lot of memory during that operation (~40MiB), but it was never an issue in previous versions of Unity.

Could something have changed under the hood in how GC is managed? Maybe in preparation for incremental GC? Is there any way to avoid this error?

2

Make that two people. It seems to occur whenever the application exceeds 3.5GB of CPU memory allocation. It’s like Unity is running in 32-bit mode somehow. This is running in editor mode.

3

We also have this happening when we recompile or try to make a build. It always happens around compilation but the crash stack trace is always different. We can repro it with only script and assembly(.dll) assets in the project. This was after we upgraded our project to 2018.3. We assumed it was something in our source that the compiler was having troubles compiling. We’ve had bugs like that before unfortunately. It takes a while to figure out what peice of code it was so we are just avoiding the upgrade for now. Though, I am not sure it is the same issue the OP is having.

4

Could you please submit a bug-report, as described in this document:

After you submitted the report, you receive a confirmation email with a Case number. Please post this bug-report Case number here (in this thread) for Unity staff to pick up.

1 Like
5

Yes, bug reports would be much appreciated.

6

I can submit a bug report but I can’t guarantee a 100% repro. It seems to depend on memory layout & usage prior to the intensive operation. Would it help more if I waited to get an actual crash from it so that the dump is included?

7

It probably does.

8

I just submitted a crash dump. Case 1104564.

1 Like
9

Thanks. It would help a lot if you could attach a repro project to the bug report, is that possible?

10

I submitted a bug report on November 15th. The case number is 1101240. I attached several logs with crash stack traces in them. I cannot provide a repro project as it would currently require us to send our entire code base.

11

Do you use Vegetation Studio? I think someone with Vegetation Studio Pro has already submitted something. If not, I would try to get a test project. In my project it is reproducible in 5 or 6 clicks.

12

Thanks for the hint! We have this bug logged for it: https://issuetracker.unity3d.com/product/unity/issues/guid/1086743/

This might be related to your cases, but it’s hard to tell for sure without looking at your projects. More reproduction projects would be very welcome.

13

I just attached a very simple repro project to my fogbugz case. It just spams allocations in an update loop until it crashes. The issue isn’t that it crashes, so much as when it does. Which is just after about 4GB of allocations occurs. Then unrelated variables reset to zero, and other weird stuff happens. I used 2018.3.0b12 for this.

2 Likes
14

Hey Darryl,

I wrote this reply on the bug itself, but since there is this forum discussion about the topic, I thought I’d share it here for others interested:

I looked at your repro project. I don’t consider this a bug, as it is a script which just allocates memory until it runs out, which it is bound to do eventually.

Now, in regards to your assumptions on when it should run out vs when it actually does: You say that you have 32 GB of RAM on your machine. But that hardly makes any difference when it comes to how much memory you can allocate.

  1. In modern operating systems with virtual memory, allocations don’t need to be tied to physical memory but can be swapped to disk. So it is actually possible to allocate more memory then you have, it will become very slow.
  2. However, when it comes to managed memory in Unity, the garbage collector keeps data structures of the memory it manages, as it needs to keep track of it. The size of these data structures limits how much memory you can allocate, and may be much less than the amount of raw memory you could allocate. Now, it would certainly be possible to change these values in the GC to allow more managed memory - but there is a tradeoff between the size of these values and performance. Bigger values would mean bigger indices into bigger data structures, making GC operations more expensive to process. The current values have been chosen to be what can be considered a reasonable tradeoff by monos developers. Now, I have not done any testing in practice as to what the performance impact of changing this would be, and there may well be valid reasons to increase them (possibly just for the editor), if we find that developers are unable to write code performing reasonable tasks within these boundaries. But a script allocating memory until it crashes does not seem like a reasonable use case for this.
  3. As for your question why your memory counter wraps around at 4GB: That’s because the counter is a uint, which is an unsigned 32-bit integer, which can only store values up to 4294967295.

jonas

15

I’ll reply to myself here - it seems I may have been a bit too fast to dismiss this. Sorry for that!

While it is correct that there are hard-coded limits in the GC, when testing this project in 2018.2 it does indeed get much further then in 18.3 - which seems to indicate that we have changed these limits (probably by accident when we merged some GC code between mono and il2cpp). This means that code could previously assume that it could use more memory, and that reducing this amount makes this a regression. I will look into what might have caused this change!

5 Likes
16

Thanks for following up. In our case, we are far from reaching even the 1GB threshold, we’re still in the MBs, so I doubt the heap is actually running out of space.

17

For the benefit of others who are hitting an absolute allocation limit: I was able to work around this issue by using NativeArrays. Of course, you have to manually dispose the memory, but that’s often acceptable.

18

FYI, this regression has been fixed and we switched the limits back to what it was in 18.2. The fix should either be in the 2018.3.0 first release or in the first patch release, depending on timing.

3 Likes
19

that’s a good news for the fix !
So I can deduce that you will release rc or maybe final soon ?
Even with all the warning in the poll feedback :confused:

20

Good news, let’s hope is coming in 2018.3.0