use a shared object with multiple length keys to store regl buffers

[bmschmidt]

Previously deepscatter had something called a TileBufferManager. This doesn't make a huge amount of sense, and had a concrete cost; if two tiles used the same underlying Arrow memory for a column (say, 2**16 floating point zeros), then we'd use different memory on the GPU for them.

This PR revamps that storage so that it uses the new TupleMap class. There is a single shared buffer manager across the whole plot, which uses a key that includes the tile itself and then the string name of the field desired (which is now actually string[], for help in the next PR.)

This map associates each column with its underlying Arrow vector memory, and it is that memory that is used as a key to the GPU buffers. This is a little weird, but the two-step operation means taht we can re-use memory more efficiently.

At some later point I'll more fully build on this. One goal is that for four types of floating point data we can always use a single webGL buffer instead of re-allocating them ad-infinitum:

1. All-false (or all-zero in float32 land)
2. All-true (or all-one in float32 land)
3. Nth-item-true, rest false (by creating a big array with a one in the middle at position K, and starting the view of that buffer at position K-N.
4. Nth-item-false, rest true (inverse of number 3).

---

Important

Refactor buffer management by replacing TileBufferManager with a shared BufferManager for efficient GPU memory usage across tiles.

• Buffer Management:
  • Replaced TileBufferManager with a shared BufferManager using TupleMap for efficient GPU memory management.
  • BufferManager uses a key with tile and field name for buffer management.
  • Introduced BufferManager.get and BufferManager.ready for buffer retrieval and readiness checks.
• Renderer Updates:
  • Updated ReglRenderer to use a single BufferManager instance.
  • Removed _buffer_manager references from Tile class.
• TileDrawProps:
  • Updated to include tile and count properties.
• Memory Efficiency:
  • Improved memory efficiency by reusing GPU buffers for common data patterns.
• Misc:
  • Removed 'observable10' from color schemes in ColorChange.svelte.

^{This description was created by for 5903dbd. It will automatically update as commits are pushed.}

[bmschmidt]

• allow subfield access #141
• use a shared object with multiple length keys to store regl buffers #140 👈
• add tuple map class #139 : 1 other dependent PR (#142 )
• Remove the deferred functions queue #138
• longer .gitignore #137
• 09 04 add ellipsis code review instructions #136
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bmschmidt and the rest of your teammates on Graphite

[ellipsis-dev]

The logic for creating a buffer for 'ix_in_tile' is repeated in both the constructor and the create_regl_buffer method. Consider refactoring to avoid repetition and adhere to the DRY principle.

[ellipsis-dev]

The convertToGlFormat method is complex and handles various data types. Consider adding comments to explain the logic, especially for timestamp conversion and dictionary handling, to improve code readability and maintainability.

[ellipsis-dev]

Using WeakMap for bufferMap is incorrect as WeakMap should be used for objects, not ArrayBufferView. Consider using a regular Map instead.

[ellipsis-dev]

The deleteColumn method does not free GPU memory, which could lead to memory leaks. Consider implementing GPU memory release when a column is deleted.

[RLesser]

These two PRs need to get rebased in some form right? I'm seeing "Trunk branch locked" in graphite

[bmschmidt]

Yeah

[bmschmidt]

It's weird though they used to be on dev which is locked, but now they're against main but graphite seems unaware.

[ellipsis-dev]

The import 'buffer' from 'd3' is unused and can be removed to clean up the code.

[bmschmidt]

Merged through other mans