Overviewer for windows crashing every time the actual render begins

[ace-savant]

Starting a few weeks ago, every time I run a render job with overviewer for Windows, it crashes with memory errors. Originally it was claiming that there was not enough virtual memory available. I have adjusted the page file drastically to try and compensate for this ranging from no page file (which should really be the case, as this server has 32GB of RAM and nothing ever comes close to saturating it), all the way to having a page file twice the size of the available RAM (64GB). No matter what, every time I run a render anywhere between one and three instances of overviewer.exe crash when the actual rendering begins. This is difficult because although the render appears to complete, the program itself never finishes updating it's tracking, which means every time run a new-render, it starts with the following:

2020-01-13 08:27:25 Welcome to Minecraft Overviewer!
2020-01-13 08:27:26 Generating textures...
2020-01-13 08:27:42 W The last render for 'day' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:27:42 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:27:42 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 21:59:29.
2020-01-13 08:28:17 Generating textures...
2020-01-13 08:28:33 W The last render for 'dayFlip' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:28:33 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:28:33 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:18.
2020-01-13 08:28:40 W The last render for 'night' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:28:40 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:28:40 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:25.
2020-01-13 08:28:46 W The last render for 'nightFlip' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:28:46 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:28:46 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:31.
2020-01-13 08:28:52 W The last render for 'caves' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:28:52 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:28:52 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:38.
2020-01-13 08:28:58 W The last render for 'cavesflip' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:28:58 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:28:58 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:44.
2020-01-13 08:29:05 W The last render for 'end' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:29:05 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:29:05 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:50.
2020-01-13 08:29:11 W The last render for 'endflip' didn't finish. I'll be scanning all the tiles to make sure everything's up to date.
2020-01-13 08:29:11 W The total tile count will be (possibly wildly) inaccurate, because I don't know how many tiles need rendering. I'll be checking them as I go.
2020-01-13 08:29:11 The unfinished render was a --forcerender. Rerendering any tiles older than 01/07/20 22:00:56.
2020-01-13 08:29:11 Preprocessing...

This causes wildly inaccurate tile counts, and it also appears to be causing overviewer to render the entire map every time I run a new job, which leads to excessive time and compute resources wasted. I would be happy to provide any additional information that can lead to this issue being resolved. I have been running the overviewer for years with no issue quite like this one and definitely would like to be part of the solution, not just a complainer.

 The error output that overviewer gives when the appcrash messages appear is below:

2020-01-13 09:25:12 Rendered 0 of 8601119400. 0% complete. ETA: ------
Process MultiprocessingDispatcherProcess-19:
Process MultiprocessingDispatcherProcess-18:
Traceback (most recent call last):
Traceback (most recent call last):
File "multiprocessing\process.py", line 297, in _bootstrap
File "overviewer_core\dispatcher.py", line 262, in run
File "overviewer_core\dispatcher.py", line 225, in update_tilesets
File "multiprocessing\managers.py", line 817, in _getvalue
File "multiprocessing\managers.py", line 796, in _callmethod
File "multiprocessing\connection.py", line 251, in recv
File "overviewer_core\textures.py", line 112, in setstate
File "overviewer_core\textures.py", line 137, in generate
MemoryError
File "multiprocessing\process.py", line 297, in _bootstrap
File "overviewer_core\dispatcher.py", line 262, in run
File "overviewer_core\dispatcher.py", line 225, in update_tilesets
File "multiprocessing\managers.py", line 817, in _getvalue
File "multiprocessing\managers.py", line 796, in _callmethod
File "multiprocessing\connection.py", line 251, in recv
File "overviewer_core\textures.py", line 112, in setstate
File "overviewer_core\textures.py", line 137, in generate
MemoryError
2020-01-13 09:27:36 Rendered 19 of 8601119400. 0% complete. ETA: 18091944h 19m 17s

The application log entry is as follows:

Faulting application name: overviewer.exe, version: 0.0.0.0, time stamp: 0x5d24a292
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000000000000
Faulting process id: 0x2018
Faulting application start time: 0x01d5ca257a23c1fa
Faulting application path: {censored}:\Minecraft Staging\overviewer-0.15.2\overviewer.exe
Faulting module path: unknown
Report Id: e5cfd4cf-3618-11ea-810e-d0509951bb49
Faulting package full name:
Faulting package-relative application ID:

The server specifications are as follows:
CPU: Intel Xeon E5-2699 v4 ES
RAM: 32GB DDR4
HDD: 256GB nVME SSD
(Other HDDs present, but I don't know that that information would be helpful or relevant)

[CounterPillow]

Okay, I can see why this is happening and it certainly puts a nail into our currently (kind of dumb) block ID approach. The blockmap is in fact a list, so as people were adding blocks with higher and higher purely invented IDs, the memory consumption of that list kept growing.

So, using a dict instead should be easy, right? Well not really. We need to index into this list in some very hot code in iterate.c. To get an item from a Python dict in C, we need to create the appropriate PyObject* from the unsigned integer that we want to use. This naturally means an object allocation, increase of refcount, then decrease of refcount with eventual deallocation, for every block drawn at least once, in the worst case twice. This is horrid.

Which means we finally need to come up with a system to dynamically generate block IDs so instead of the ridiculously sparse list we have right now, we get something tight that can be indexed in a fast way.