[FEAT] connect: createDataFrame

[andrewgazelka]

No description provided.

[andrewgazelka]

• [FEAT] connect: createDataFrame #3363 👈 (View in Graphite)
• [FEAT] connect: support basic column operations #3362 : 1 other dependent PR (#3350 )
• [FEAT] connect: to_daft_* use ref instead of value #3355 : 2 other dependent PRs (#3357 , #3360 )
• [FEAT] connect: add alias support #3342 : 6 other dependent PRs (#3344 , #3345 , #3348 and 3 others)
• [FEAT] consolidate Spark session fixture into conftest.py #3341 : 3 other dependent PRs (#3309 , #3326 , #3343 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

CodSpeed Performance Report

Merging #3363 will degrade performances by 66.8%

_{Comparing andrew/connect-create-dataframe (69b2c29) with main (d1d0fab)}

Summary

⚡ 1 improvements
❌ 1 regressions
✅ 15 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	andrew/connect-create-dataframe	Change
❌	test_iter_rows_first_row[100 Small Files]	107.5 ms	323.8 ms	-66.8%
⚡	test_show[100 Small Files]	23.9 ms	16 ms	+48.71%

[codecov]

Codecov Report

Attention: Patch coverage is 57.83133% with 175 lines in your changes missing coverage. Please review.

Project coverage is 77.29%. Comparing base (de4fe50) to head (69b2c29).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-connect/src/translation/datatype/codec.rs	47.33%	89 Missing ⚠️
...ect/src/translation/logical_plan/local_relation.rs	61.01%	69 Missing ⚠️
daft/dataframe/dataframe.py	8.33%	11 Missing ⚠️
...-connect/src/translation/logical_plan/aggregate.rs	0.00%	5 Missing ⚠️
...daft-connect/src/translation/logical_plan/to_df.rs	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3363      +/-   ##
==========================================
- Coverage   77.35%   77.29%   -0.06%     
==========================================
  Files         696      699       +3     
  Lines       84849    85242     +393     
==========================================
+ Hits        65631    65887     +256     
- Misses      19218    19355     +137     

Files with missing lines	Coverage Δ
src/daft-connect/src/lib.rs	66.34% <100.00%> (+2.13%)	⬆️
src/daft-connect/src/op/execute/root.rs	95.45% <100.00%> (-0.20%)	⬇️
src/daft-connect/src/translation/datatype.rs	20.81% <ø> (+10.40%)	⬆️
src/daft-connect/src/translation/logical_plan.rs	80.64% <100.00%> (+9.21%)	⬆️
...ft-connect/src/translation/logical_plan/project.rs	88.88% <100.00%> (ø)
...daft-connect/src/translation/logical_plan/range.rs	62.50% <100.00%> (ø)
src/daft-connect/src/translation/schema.rs	100.00% <ø> (ø)
src/daft-local-execution/src/pipeline.rs	94.48% <100.00%> (+0.02%)	⬆️
src/daft-logical-plan/src/builder.rs	91.66% <ø> (ø)
...daft-connect/src/translation/logical_plan/to_df.rs	94.73% <94.73%> (ø)
... and 4 more

... and 5 files with indirect coverage changes

[universalmind303]

Most of the feedback can be addressed later. But I definitely think we should remove the arrow_format dependency before merging.

Also the with_columns -> select change is needed before merging