Skip to content

Commit

Permalink
Migrate YDB Operator to new DBAPI
Browse files Browse the repository at this point in the history
  • Loading branch information
vgvoleg committed Nov 7, 2024
1 parent 995cd8f commit 7719e97
Show file tree
Hide file tree
Showing 16 changed files with 43 additions and 1,086 deletions.
3 changes: 2 additions & 1 deletion docs/apache-airflow-providers-ydb/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ PIP package Version required
======================================= ==================
``apache-airflow`` ``>=2.8.0``
``apache-airflow-providers-common-sql`` ``>=1.14.1``
``ydb`` ``>=3.12.1``
``ydb`` ``>=3.18.8``
``ydb-dbapi`` ``>=0.1.0``
======================================= ==================

Cross provider package dependencies
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ workflow. Airflow is essentially a graph (Directed Acyclic Graph) made up of tas
A task defined or implemented by a operator is a unit of work in your data pipeline.

The purpose of this guide is to define tasks involving interactions with a YDB database with
the :class:`~airflow.providers.ydb.operators.YDBExecuteQueryOperator` and :class:`~airflow.providers.ydb.operators.YDBScanQueryOperator`.
the :class:`~airflow.providers.ydb.operators.YDBExecuteQueryOperator`.

Common database operations with YDBExecuteQueryOperator
-------------------------------------------------------
Expand Down Expand Up @@ -162,26 +162,6 @@ by creating a sql file.
)
Executing Scan Queries with YDBScanQueryOperator
-------------------------------------------------------

YDBScanQueryOperator executes YDB Scan Queries, which designed primarily for running analytical ad hoc queries. Parameters of the operators are:

- ``sql`` - string with query;
- ``conn_id`` - YDB connection id. Default value is ``ydb_default``;
- ``params`` - parameters to be injected into query if it is Jinja template, more details about :doc:`params <apache-airflow:core-concepts/params>`

Example of using YDBScanQueryOperator:

.. code-block:: python
get_birth_date_scan = YDBScanQueryOperator(
task_id="get_birth_date_scan",
sql="sql/birth_date.sql",
params={"begin_date": "2020-01-01", "end_date": "2020-12-31"},
)
The complete YDB Operator DAG
-----------------------------

Expand All @@ -196,7 +176,7 @@ When we put everything together, our DAG should look like this:
Conclusion
----------

In this how-to guide we explored the Apache Airflow YDBExecuteQueryOperator and YDBScanQueryOperator to connect to YDB database. Let's quickly highlight the key takeaways.
In this how-to guide we explored the Apache Airflow YDBExecuteQueryOperator to connect to YDB database. Let's quickly highlight the key takeaways.
It is best practice to create subdirectory called ``sql`` in your ``dags`` directory where you can store your sql files.
This will make your code more elegant and more maintainable.
And finally, we looked at the templated version of sql script and usage of ``params`` attribute.
3 changes: 2 additions & 1 deletion generated/provider_dependencies.json
Original file line number Diff line number Diff line change
Expand Up @@ -1395,7 +1395,8 @@
"deps": [
"apache-airflow-providers-common-sql>=1.14.1",
"apache-airflow>=2.8.0",
"ydb>=3.12.1"
"ydb-dbapi>=0.1.0",
"ydb>=3.18.8"
],
"devel-deps": [],
"plugins": [],
Expand Down
16 changes: 0 additions & 16 deletions providers/src/airflow/providers/ydb/hooks/_vendor/__init__.py

This file was deleted.

This file was deleted.

194 changes: 0 additions & 194 deletions providers/src/airflow/providers/ydb/hooks/_vendor/dbapi/connection.py

This file was deleted.

Loading

0 comments on commit 7719e97

Please sign in to comment.