Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(catalog): Cutover deprecated APIs to use session, catalog, table abstractions [3/3] #3830

Merged
merged 22 commits into from
Feb 25, 2025

Conversation

rchowell
Copy link
Contributor

@rchowell rchowell commented Feb 19, 2025

This PR swaps existing functionality to be backed by the new session, catalog, and table APIs.

Changes

  • Adds set_namespace and current_namespace for qualified name resolution control
  • Adds support for catalog-qualified and schema-qualified identifiers
  • Adds the new session APIs to daft.* top-level
    • daft.register_python_catalog -> daft.attach_catalog
    • daft.unregister_catalog -> daft.detach_catalog
    • daft.read_table -> daft.read_table (via session)
    • daft.register_table -> daft.attach_table
  • Ports existing rust tests for DaftMetaCatalog to the Session.

Context

@github-actions github-actions bot added the feat label Feb 19, 2025
@rchowell rchowell changed the base branch from main to rchowell/catalog-2-of-3 February 19, 2025 21:46
@rchowell rchowell force-pushed the rchowell/catalog-3-of-3 branch from e4725d7 to c47b29e Compare February 19, 2025 22:13
@rchowell rchowell marked this pull request as ready for review February 19, 2025 23:47
@rchowell rchowell force-pushed the rchowell/catalog-2-of-3 branch from f1a2a5a to 87810f1 Compare February 21, 2025 00:25
@rchowell rchowell force-pushed the rchowell/catalog-2-of-3 branch from 7165ed8 to 470d238 Compare February 21, 2025 15:22
Base automatically changed from rchowell/catalog-2-of-3 to main February 21, 2025 18:33
@kevinzwang
Copy link
Member

Looks like there's a lot of merge conflicts. Could you merge from main first?

@rchowell rchowell force-pushed the rchowell/catalog-3-of-3 branch from 91957d9 to 1a60f1d Compare February 25, 2025 01:05
Copy link

codecov bot commented Feb 25, 2025

Codecov Report

Attention: Patch coverage is 76.89970% with 76 lines in your changes missing coverage. Please review.

Project coverage is 78.06%. Comparing base (0098bee) to head (8b8be7f).
Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
daft/session.py 67.16% 22 Missing ⚠️
src/daft-session/src/python.rs 21.05% 15 Missing ⚠️
src/daft-session/src/session.rs 86.86% 13 Missing ⚠️
daft/catalog/__init__.py 66.66% 11 Missing ⚠️
src/daft-catalog/src/error.rs 0.00% 8 Missing ⚠️
daft/catalog/__unity.py 25.00% 3 Missing ⚠️
src/daft-catalog/src/python.rs 88.88% 2 Missing ⚠️
daft/__init__.py 0.00% 1 Missing ⚠️
daft/catalog/__iceberg.py 75.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3830      +/-   ##
==========================================
+ Coverage   77.92%   78.06%   +0.14%     
==========================================
  Files         763      760       -3     
  Lines       95766    95865      +99     
==========================================
+ Hits        74626    74840     +214     
+ Misses      21140    21025     -115     
Files with missing lines Coverage Δ
src/daft-catalog/src/catalog.rs 0.00% <ø> (ø)
src/daft-catalog/src/identifier.rs 97.52% <100.00%> (+17.16%) ⬆️
src/daft-catalog/src/table.rs 74.19% <100.00%> (+2.76%) ⬆️
src/daft-sql/src/planner.rs 80.42% <100.00%> (+0.07%) ⬆️
src/daft-sql/src/python.rs 55.40% <ø> (ø)
src/lib.rs 96.15% <ø> (-0.15%) ⬇️
daft/__init__.py 23.07% <0.00%> (ø)
daft/catalog/__iceberg.py 76.08% <75.00%> (+17.39%) ⬆️
src/daft-catalog/src/python.rs 77.77% <88.88%> (+11.11%) ⬆️
daft/catalog/__unity.py 48.27% <25.00%> (ø)
... and 5 more

... and 9 files with indirect coverage changes

Copy link
Member

@kevinzwang kevinzwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Am I correct to think that we're working on getting qualified table support in SQL as well, and that we will eventually eliminate SQLCatalog and only use the session catalog everywhere and for all our frontends?

}
}
pub curr_catalog: Option<String>,
pub curr_namespace: Option<Vec<String>>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see we're moving away from the term "namespace" and into "qualifier" in some places but not in others. How are you thinking about the difference between these two terms?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The qualifier can contain a catalog, so it's not exactly a namespace. Namespace is like a dir in a volume if the volume is catalog.

@rchowell rchowell merged commit 5db17c2 into main Feb 25, 2025
43 of 44 checks passed
@rchowell rchowell deleted the rchowell/catalog-3-of-3 branch February 25, 2025 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants