🐙 source-twilio: per partition states for nested streams

[mkrawc]

What

• Manage per partition state for nested streams
• Get Usage records per day so that usage can be analyzed and aggregated
• Fix edge case that Usage endpoint may return 404 if requesting usage for start/end dates way before the account creation date

How

• use StreamSlice to manage stream slices and partitions
• calculate UsageRecords min datetime based on accounts date_created

Review guide

User Impact

• After upgrade existing connectors will download all data since the configured start_date because state format changed

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2024 1:12pm

[vercel]

@mkrawc is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.

[marcosmarxm]

Hello @mkrawc is there an open issue reporting the problem?

[marcosmarxm]

@btkcodedev do you mind taking a look into this contribution

[mkrawc]

Hello @mkrawc is there an open issue reporting the problem?

@marcosmarxm I don't think so. I was working with twilio connector and noticed those issues.

The main problem is that if the stream state is saved globally across all accounts the connector may miss data for subaccounts. So the scenario is:

• connector lists accounts
• connector downloads the first account data for requestes timeframe
• connector sets the state to the latest downloaded date
• connector crashes (or gets cancelled) before downloading data for all of the subaccounts
• connector restarts with the stream state pointing to the latest date
• connector doesn't read the data for remaining subaccounts

After the change the connector state is saved per individual subaccounts for after the failure the connector would pick up where it ended last time.

The other two changes are:

• Get Usage records per day so that usage can be analyzed and aggregated - Before the change the connector would save just one usage record since the connector start date to the current date which makes the date useless for further aggregations. After the change the connector will save usage records for each day so that the data can be summed for a desired timeframe.
• Fix edge case that Usage endpoint may return 404 if requesting usage for start/end dates way before the account creation date - It looks like Twilio API returns 404 errors when requesting usage records where both StartDate and EndDate are before when the account was created:

404 Client Error: Not Found for url: https://api.twilio.com/2010-04-01/Accounts/xxxxx/Usage/Records/Daily.json?PageSize=1000&StartDate=2022-12-01&EndDate=2023-12-01

Hope this helps. Let me know if you have any other questions.

[btkcodedev]

Looking forward to this

[btkcodedev]

Commenting on breaking change. If the spec change is important, we need a review from breaking-change-reviewers too

[btkcodedev]

If there is a spec change, it is a breaking change. We need to bump the version to 1.0.0 and add a twilio-migrations.md file.
But first, do we really need to change this?

[mkrawc]

@btkcodedev I don't think this change is critical. Account ID doesn't sound like a secret variable and just thought it doesn't need to be a secret. I'll revert it to not cause a breaking change.