Merge sync and async operator #40
Conversation
pankajastro
force-pushed
the
merge_sync_async_op
branch
from
0846400
to
605b6f5
Compare
There was a problem hiding this comment.
The repo name is airflow-provider-fivetran-async and we are deprecating the async operator. I think we should merge this the other way, i.e., having FivetranOperatorAsync and have deferrable as True by default. Having it as False, should invoke the sync part of the operator
The current default value of the deferrable param is True. I decided to keep FivetranOperator without the suffix Async because I was thinking of having the suffix Async in the operator/sensor name and the deferrable param looks not in sync. But happy to change it if you think otherwise makes more sense. |
fivetran_provider_async/example_dags/example_fivetran_bigquery.py
Outdated
Show resolved
Hide resolved
pankajastro
force-pushed
the
merge_sync_async_op
branch
2 times, most recently
from
9221e83
to
3b0de44
Compare
pankajastro
force-pushed
the
merge_sync_async_op
branch
from
fef66ec
to
39bfa16
Compare
phanikumv
dismissed
their stale review
agree with the point discussed on sync vs async merge
| `FivetranSensorAsync` allows you to monitor a Fivetran sync job for completion before running downstream processes. | ||
| `FivetranOperatorAsync` submits a Fivetran sync job and polls for its status on the triggerer. | ||
| `FivetranSensor` allows you to monitor a Fivetran sync job for completion before running downstream processes. | ||
| `FivetranOperator` submits a Fivetran sync job and polls for its status on the triggerer. |
There was a problem hiding this comment.
| `FivetranOperator` submits a Fivetran sync job and polls for its status on the triggerer. | |
| `FivetranOperator` submits a Fivetran sync job and polls for it's status on the triggerer.When deferrable param is set to False, it would not release the Airflow worker slot and waits for the job to complete. |
There was a problem hiding this comment.
@phanikumv the FivetranOperator does not wait for the job to complete. It only submits and then we use a sensor to monitor for completion
There was a problem hiding this comment.
ah right sorry , just confused with OSS operators :)
closes: #35