-
Notifications
You must be signed in to change notification settings - Fork 414
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: remove the get_data_catalog() function #1941
fix: remove the get_data_catalog() function #1941
Conversation
This is related to delta-io#1713 and delta-io#1860 insofar that it removes the `get_data_catalog()` function. This is being done separately from the sub-crates work since it's easy enough to remove and frankly doesn't work properly anyways for Python users, which is the only caller we currently have in our codebase Fixes delta-io#1860
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
It was also quite limited in functionality.. for example unity catalog basically meant you still need storage level access which in most situations you won't get. @rtyler are we going to plan to support this in the future again? Before we can merge we probably should add a bunch of deprecation messages in Python since it was working before at least for unity catalog |
@ion-elgreco I dunno, I strongly doubt people are using that functionality. The Unity catalog functionality is practically worthless IMHO since you have to have credentials for Unity catalog to lookup a table's location, and then a second set of credentials to actually access that storage. IMHO it is not worth slowing changes for a couple reasons:
I however defer to @wjones127 here since he's much more familiar with the Python ecossytem than I am. I do need this to merge before #1825 because there's this presumption in this API that all available catalogs will be compiled into deltalake-core which is I need to remove for sub-crates. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
as you mentioned the catalogs in python are somewhat non functional right now anyhow ...
Given they are not functional and don't have a maintainer right now, I'm fine removing them for now. |
We need to remove the python code paths as well. |
I'm trying/struggling to understand the current state of Unity Catalog support.
|
@dhirschfeld I would love to get UC support but this is all in the hands of databricks. They simply made it impossible to read data that is behind a catalog outside of databricks... |
Well, that's disappointing 😞
...but maybe that just gives access from within the DataBricks UI and doesn't actually expose the underlying storage token 😞 |
@dhirschfeld One of the requirements of grabbing the storage credential is: But I am not sure that would be the same as read access to a certain table, so you may still need some form of high level access, and I assume the people who have access to the managed credential they likely already have access directly to the object store. If you find another way, we can definetely add it! :) I'm personally also hampered by the fact that I can't read delta tables in UC with delta-rs |
I don't know much about Databricks, and less about Unity Catalog, but I'll have a poke about to see what I can find. If you have been unable to find a way to do it, I'm not that hopeful, but we'll see... I opened #2066 to track my progress! 😄 |
This is related to #1713 and #1860 insofar that it removes the
get_data_catalog()
function. This is being done separately from the sub-crates work since it's easy enough to remove and frankly doesn't work properly anyways for Python users, which is the only caller we currently have in our codebaseFixes #1860