Migrate YDB Operator to new DBAPI #43784
Merged
+47
−1,086
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vgvoleg
force-pushed
the
update_ydb_provider
branch
from
ab9272e
to
68d717b
Compare
vgvoleg
changed the title
dabla
suggested changes
Nov 7, 2024
vgvoleg
force-pushed
the
update_ydb_provider
branch
3 times, most recently
from
9725aff
to
7719e97
Compare
vgvoleg
changed the title
potiuk
reviewed
Nov 11, 2024
potiuk
approved these changes
Nov 11, 2024
|
Nice set of changes BTW :) |
potiuk
approved these changes
Nov 11, 2024
vgvoleg
force-pushed
the
update_ydb_provider
branch
from
2d2084c
to
59a63b3
Compare
uzhastik
reviewed
Nov 11, 2024
vgvoleg
force-pushed
the
update_ydb_provider
branch
from
0621048
to
3df65fd
Compare
vgvoleg
force-pushed
the
update_ydb_provider
branch
from
3df65fd
to
3fa7ac3
Compare
uzhastik
approved these changes
Nov 12, 2024
sunank200
pushed a commit
to astronomer/airflow
that referenced
this pull request
Nov 12, 2024
* Migrate YDB Operator to new DBAPI * Update changelog
eladkal
reviewed
Nov 12, 2024
Comment on lines
+30
to
+33
| .. note:: | ||
| This release removes YDBScanQueryOperator from this provider package. | ||
| At this point, YDBExecuteQueryOperator could load unlimited amount of rows, so no specific operator is needed. | ||
|
|
There was a problem hiding this comment.
is this a breaking change?
There was a problem hiding this comment.
Yes, we've deleted operator that was added in prev release. I'm pretty sure that no customers used it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For historical reasons, the YDB operator had a copied DBAPI from the ydb-platform/ydb-sqlalchemy repository. It was impossible to use it without copying - the repository has a dependency on SQLAlchemy > 2.0. (problem was described here https://github.com/apache/airflow/blob/995cd8fba8de4ab3d041c7e141a0b92d786a8277/providers/src/airflow/providers/ydb/hooks/_vendor/readme.md)
At the moment we split the DBAPI and SQLAlchemy Dialect, so now there are no conflicts between them.
In this pull request, we delete the copied files, adding a new dependency on the ydb-dbapi package, and also adapt the current YDB Operator to the new dbapi.
YDBScanQueryOperator was more of an experiment, the new YDBExecuteQueryOperator on top of the new dbapi covers all scenarios - no need to stop it (as far as I know, none of our users have started using it)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.