Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce SortedDataSelection class #144

Merged
merged 3 commits into from
Sep 20, 2024

Conversation

bmschmidt
Copy link
Collaborator

@bmschmidt bmschmidt commented Sep 16, 2024

introduce SortedDataSelection class using a new pivot strategy

SortedDataSelection uses a lot more memory than a selection but enables getting the k-highest element quickly (roughly O(log N) time).

I went back a bunch with the fancy new GPT-4 chain of thought to make this.

Includes tests.

The next PR in this stack will allow faster in-order iteration over a list using a minheap structure.


🚀 This description was created by cb0bce7

feat: add SortedDataSelection class for efficient data sorting and selection

Summary:

Introduce SortedDataSelection class for efficient data sorting and selection with tests and future enhancements.

Key points:

  • Introduces SortedDataSelection class for efficient k-highest element retrieval using a pivot strategy.
  • Supports sorting by numeric fields and quick selection using Quickselect algorithm.
  • Updates DataSelection to support new sorting functionality.
  • Adds SelectionTile class to manage tile-specific sorting data.
  • Updates exports in deepscatter.ts to include SortedDataSelection.
  • Adds tests for SortedDataSelection in dataset.spec.js.
  • Adds publish_docs_if_stable.sh script for documentation publishing.
  • Minor type and function adjustments in utilityFunctions.ts.
  • Uses more memory but enables quick k-highest element retrieval.
  • Future PR will allow faster in-order iteration using a minheap structure.

Generated with ❤️ by ellipsis.dev

Copy link
Collaborator Author

bmschmidt commented Sep 16, 2024

@bmschmidt bmschmidt marked this pull request as ready for review September 16, 2024 19:52
@bmschmidt bmschmidt requested a review from eelegiap September 16, 2024 19:52
Copy link

@eelegiap eelegiap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can follow everything except the QuickSelect. I was confused about what comparisonGetter/sortOperation does.

@bmschmidt bmschmidt force-pushed the 08-07-make_the_tiles_on_a_selection_a_struct_of_info_not_twinned_lists branch from b59865f to 6fda8c1 Compare September 19, 2024 19:00
@bmschmidt bmschmidt force-pushed the 08-08-introduce_sorteddataselection_class branch from 3f1fa61 to 00ccb8b Compare September 19, 2024 19:00
Copy link
Collaborator Author

bmschmidt commented Sep 19, 2024

Merge activity

@bmschmidt bmschmidt force-pushed the 08-07-make_the_tiles_on_a_selection_a_struct_of_info_not_twinned_lists branch from 6fda8c1 to 3c779e2 Compare September 20, 2024 02:56
@bmschmidt bmschmidt force-pushed the 08-08-introduce_sorteddataselection_class branch from 00ccb8b to 39c3fa1 Compare September 20, 2024 02:57
let t;
indices: T,
): Vector<Dictionary<Utf8, ArrayToArrayMap[T]>> {
let t: ArrayToArrayMap[T];

if (indices[Symbol.toStringTag] === `Int8Array`) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use instanceof for type checking instead of Symbol.toStringTag.

Suggested change
if (indices[Symbol.toStringTag] === `Int8Array`) {
if (indices instanceof Int8Array) {

@bmschmidt bmschmidt changed the base branch from 08-07-make_the_tiles_on_a_selection_a_struct_of_info_not_twinned_lists to graphite-base/144 September 20, 2024 03:16
@bmschmidt bmschmidt changed the base branch from graphite-base/144 to main September 20, 2024 03:18
@bmschmidt bmschmidt force-pushed the 08-08-introduce_sorteddataselection_class branch from 39c3fa1 to cb0bce7 Compare September 20, 2024 03:20
@bmschmidt bmschmidt merged commit 8b458b7 into main Sep 20, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants