Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Donate Ray SQL to the DataFusion Project #1

Merged
merged 45 commits into from
Sep 30, 2024
Merged

Donate Ray SQL to the DataFusion Project #1

merged 45 commits into from
Sep 30, 2024

Commits on Sep 20, 2024

  1. Initial commit

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    b450f56 View commit details
    Browse the repository at this point in the history

  2. Basic project structure

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    7700662 View commit details
    Browse the repository at this point in the history

  3. gitignore

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d8693d3 View commit details
    Browse the repository at this point in the history

  4. Add protobuf plumbing (apache#2)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    76d5910 View commit details
    Browse the repository at this point in the history

  5. Implement protobuf codec

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    e10dcc6 View commit details
    Browse the repository at this point in the history

  6. Wire up query execution (apache#4)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    46eb485 View commit details
    Browse the repository at this point in the history

  7. query runs end to end (apache#5)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    852967a View commit details
    Browse the repository at this point in the history

  8. re-organize python code (apache#6)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f5d52db View commit details
    Browse the repository at this point in the history

  9. Implement shuffle more fully (apache#7)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    94ed962 View commit details
    Browse the repository at this point in the history

  10. update README (apache#8)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    34ac119 View commit details
    Browse the repository at this point in the history

  11. Bug fix (apache#9)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    370b68f View commit details
    Browse the repository at this point in the history

  12. Support multiple shuffle partitions (apache#10)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f697760 View commit details
    Browse the repository at this point in the history

  13. More shuffle fixes (apache#11)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    bf60072 View commit details
    Browse the repository at this point in the history

  14. fix readme (apache#12)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    974fe1a View commit details
    Browse the repository at this point in the history

  15. add perf chart (apache#13)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    16e735a View commit details
    Browse the repository at this point in the history

  16. Remove hard-coded temp dir (apache#14)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2177b2c View commit details
    Browse the repository at this point in the history

  17. bug fix (apache#15)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5f6af95 View commit details
    Browse the repository at this point in the history

  18. New results (apache#16)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5014b5a View commit details
    Browse the repository at this point in the history

  19. Upgrade to DataFusion 17, fix a couple of bugs, add some tests (apach… 

    …e#18)
    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3c51a8a View commit details
    Browse the repository at this point in the history

  20. Remove debug logging (apache#19)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8326ea0 View commit details
    Browse the repository at this point in the history

  21. update README (apache#21)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6baa2f8 View commit details
    Browse the repository at this point in the history

  22. Make better use of futures (apache#23)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5c6760f View commit details
    Browse the repository at this point in the history

  23. Documentation & bug fixes (apache#24)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ca6e104 View commit details
    Browse the repository at this point in the history

  24. Update README.md

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    b233f08 View commit details
    Browse the repository at this point in the history

  25. [WIP] Use Ray object store for shuffle exchange (apache#28)

    @frank-lsf @austin362667
    frank-lsf authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    061c865 View commit details
    Browse the repository at this point in the history

  26. Fixes for Ray-based shuffle (apache#29)

    @frank-lsf @austin362667
    frank-lsf authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    733e949 View commit details
    Browse the repository at this point in the history

  27. Small fixes for Context (apache#30)

    @frank-lsf @austin362667
    frank-lsf authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    db9919c View commit details
    Browse the repository at this point in the history

  28. Make distributed execution work (apache#33) 

    * Make distributed execution work

* fix tips.py

* fixes; incorporate changes from apache#32
    @frank-lsf @austin362667
    frank-lsf authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    e3eb530 View commit details
    Browse the repository at this point in the history

  29. Upgrade to DataFusion 20 (apache#31)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1da4df3 View commit details
    Browse the repository at this point in the history

  30. Add support for DDL statements, such as CREATE VIEW (apache#35) 

    * Experimenting with supporting DDL

* update docs
    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8bb1742 View commit details
    Browse the repository at this point in the history

  31. Use PyArrow for zero-copy interaction with the Ray Object Store (apac… 

    …he#36)

* Optimize Ray shuffle with zero-copy object store

* remove more clones

* change bytes to pyarrow.array

* revert /tmp

* remove empty_result_set

* remove empty_result_set

* Fix input partition count bug
    @frank-lsf @austin362667
    frank-lsf authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    66b2bcf View commit details
    Browse the repository at this point in the history

  32. Add Frank as author (apache#37)

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    dedb93c View commit details
    Browse the repository at this point in the history

  33. fix hyperlink of issue 22 in docs/README.txt (apache#40) 

    Co-authored-by: ivanfan <[email protected]>
    @Ivankings @austin362667
    2 people authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    7610d97 View commit details
    Browse the repository at this point in the history

  34. delta lake and iceberg table support (apache#43) 

    * delta support

* imports
    @raviranak @austin362667
    raviranak authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    654838c View commit details
    Browse the repository at this point in the history

  35. Update DataFusion version to 28.0.0 (apache#41) 

    * Update DataFusion version

* update example
    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    3534686 View commit details
    Browse the repository at this point in the history

  36. Upgrade to DataFusion 33 (apache#45) 

    * Upgrade to DataFusion 33

* undo release profile change
    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9686879 View commit details
    Browse the repository at this point in the history

  37. Add basic GitHub workflow to compile code (apache#47) 

    * Create rust.yml

* install protobuf
    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6b09d86 View commit details
    Browse the repository at this point in the history

  38. fix

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ed38271 View commit details
    Browse the repository at this point in the history

  39. fix

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6f27a40 View commit details
    Browse the repository at this point in the history

  40. fix

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    29bdcb1 View commit details
    Browse the repository at this point in the history

  41. fix

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    cb15e73 View commit details
    Browse the repository at this point in the history

  42. fix

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    c24fa80 View commit details
    Browse the repository at this point in the history

  43. fix

    @andygrove @austin362667
    andygrove authored and austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    34208b6 View commit details
    Browse the repository at this point in the history

  44. Add ASF license header 

    Signed-off-by: Austin Liu <[email protected]>
    @austin362667
    austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    18cd940 View commit details
    Browse the repository at this point in the history

  45. Remove ASF header for generated code 

    Signed-off-by: Austin Liu <[email protected]>
    @austin362667
    austin362667 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    15e2162 View commit details
    Browse the repository at this point in the history