FivetranOperatorAsync waits forever in case of reschedule

[joshzana]

We ran into a case where we triggered a Fivetran sync of Hubspot and it hit a Hubspot API error:

From the fivetran logs:

{
  "event" : "warning",
  "data" : {
    "type" : "retry_api_call",
    "message" : "Retrying last failed API call after 78 seconds, Reason :HTTP 500 Internal Server Error"
  },
  "created" : "2023-03-20T20:01:14.545Z",
  "connector_type" : "hubspot",
  "connector_id" : "feeling_against",
  "connector_name" : "hubspot_fivetran",
  "sync_id" : "c316efc5-92a6-46a9-b759-60dbe5972b27"
}

and later:

{
  "event" : "status",
  "data" : {
    "status" : "RESCHEDULED",
    "reason" : "Rescheduled for 2023-03-20T20:54:53.894Z, reason: Rescheduled the sync due to server error",
    "rescheduledAt" : "2023-03-20T20:54:54.041Z"
  },
  "created" : "2023-03-20T20:25:47.512Z",
  "connector_type" : "hubspot",
  "connector_id" : "feeling_against",
  "connector_name" : "hubspot_fivetran",
  "sync_id" : "c316efc5-92a6-46a9-b759-60dbe5972b27"
}

According to fivetran docs:

Fivetran reschedules syncs in two situations:
Fivetran has reached the request limit for your application's API.
Fivetran has exceeded the set number of API calls we have provisioned for your connector.
In either rescheduling scenario, the sync automatically pauses and resumes the following day, once the daily API limit has > been restored.

In our case, the triggerer on our end just kept polling forever. Almost a day later, we noticed this, and cancelled the run, letting the next run trigger again.

It would be better if the fivetran sensor failed the task right away, and then we could use our own retry or schedule logic to decide what to do from there.

[pbhuss]

Related PR on the package the async operator is derived from
fivetran/airflow-provider-fivetran#69

[LukaDevic]

Partial implementation can be found in package from which provider is derived from:
https://github.com/fivetran/airflow-provider-fivetran/pull/76/files

[phanikumv]

@sunank200 can you please work on this during next week

[joshzana]

Thank you for fixing! Any idea when we can get a pypi release with this fix in it?

[phanikumv]

@joshzana we plan to do a release tomorrow

[sunank200]

@joshzana we just did a release now. Please check:https://pypi.org/project/airflow-provider-fivetran-async/