refactor: create a rust based PartitionSet

[universalmind303]

most of this is ported from the python impl inside daft/runners/partitioning.py.

Note for reviewer.

For context around why this is needed. The DataFrame class uses PartitionSet extensively for various common operations such as show, and collect. In order to add this functionality to our spark connect implementation, we need a similar construct in rust.

Ideally, I'd like to port over the python implementation to use this new rust one, but there are still a few things that I'm not entirely sure how to implement (such as RayPartitionSet)

Not all of the methods inside partitioning.rs are used yet, But I intend to follow up this PR with an implementation for #3498, and this is a prerequisite as show relies on get_preview_micropartitions.

[codspeed-hq]

CodSpeed Performance Report

Merging #3515 will improve performances by 21.32%

_{Comparing universalmind303:rust-pset (6a702a8) with main (ad175ae)}

Summary

⚡ 1 improvements
✅ 16 untouched benchmarks

Benchmarks breakdown

	Benchmark	main	universalmind303:rust-pset	Change
⚡	test_iter_rows_first_row[100 Small Files]	187.6 ms	154.6 ms	+21.32%

[codecov]

Codecov Report

Attention: Patch coverage is 25.18519% with 101 lines in your changes missing coverage. Please review.

Project coverage is 77.51%. Comparing base (9739bb6) to head (6a702a8).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-micropartition/src/partitioning.rs	17.39%	95 Missing ⚠️
src/daft-connect/src/translation/logical_plan.rs	14.28%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3515      +/-   ##
==========================================
+ Coverage   77.50%   77.51%   +0.01%     
==========================================
  Files         703      709       +6     
  Lines       85645    86279     +634     
==========================================
+ Hits        66378    66880     +502     
- Misses      19267    19399     +132     

Files with missing lines	Coverage Δ
...ect/src/translation/logical_plan/local_relation.rs	61.66% <100.00%> (+0.64%)	⬆️
src/daft-local-execution/src/pipeline.rs	93.76% <100.00%> (-0.73%)	⬇️
src/daft-local-execution/src/run.rs	89.52% <100.00%> (+0.05%)	⬆️
src/daft-local-execution/src/sources/in_memory.rs	81.25% <100.00%> (-2.09%)	⬇️
src/daft-micropartition/src/lib.rs	50.00% <ø> (ø)
src/daft-micropartition/src/micropartition.rs	90.81% <ø> (ø)
src/daft-connect/src/translation/logical_plan.rs	73.17% <14.28%> (-12.55%)	⬇️
src/daft-micropartition/src/partitioning.rs	17.39% <17.39%> (ø)

... and 75 files with indirect coverage changes

[andrewgazelka]

LGTM.

[andrewgazelka]

preference (specifically @universalmind303) on reviewers auto merging if review is requested and we ✅?

[universalmind303]

preference (specifically @universalmind303) on reviewers auto merging if review is requested and we ✅?

i generally have a preference for leaving it to the author to merge.