Code

WIP: asyncio-based CDK#34424

Code

Closed

clnoll wants to merge 74 commits intomasterairbytehq/airbyte:masterfrom asyncio-cdkairbytehq/airbyte:asyncio-cdkCopy head branch name to clipboard

+8,035-366

Commits

Commits on Jan 6, 2024

• Working: new async_http.py module

  clnoll

  committed
  e6d93ce

  Copy full SHA for e6d93ce

• Working: new file for async exceptions (some overlap with existing)

  clnoll

  committed
  8793464

  Copy full SHA for 8793464

• file rename

  clnoll

  committed
  3e39da7

  Copy full SHA for 3e39da7

• Working: new core_async.py with AsyncStream

  clnoll

  committed
  e979302

  Copy full SHA for e979302

• Working: stream_helper_async; not sure if needed

  clnoll

  committed
  bb5f26b

  Copy full SHA for bb5f26b

• Working: AsyncAbstractSource

  clnoll

  committed
  ee29b91

  Copy full SHA for ee29b91

• Working: AsyncAbstractStream

  clnoll

  committed
  d138bcf

  Copy full SHA for d138bcf

• Working: AsyncDefaultStream

  clnoll

  committed
  14fe32b

  Copy full SHA for 14fe32b

• Working: AsyncPartitionGenerator; not used

  clnoll

  committed
  6fb1eea

  Copy full SHA for 6fb1eea

• Not used: async_call_rate and rate_limiting_async

  clnoll

  committed
  65e8241

  Copy full SHA for 65e8241

• Working: async adapters for stream

  clnoll

  committed
  83a8126

  Copy full SHA for 83a8126

• Working: async Salesforce

  clnoll

  committed
  48a725d

  Copy full SHA for 48a725d

• Code for local testing

  clnoll

  committed
  52ca102

  Copy full SHA for 52ca102

• Working but not really used: AsyncPartition

  clnoll

  committed
  eef1b4f

  Copy full SHA for eef1b4f

• Use AsyncStream instead of Stream

  clnoll

  committed
  a6493ea

  Copy full SHA for a6493ea

• Working: use Limiter sessions (but need to test limiters)

  clnoll

  committed
  37b049c

  Copy full SHA for 37b049c

• Add json header if sending json data

  clnoll

  committed
  d928fff

  Copy full SHA for d928fff

• Update request args

  clnoll

  committed
  aaf19cf

  Copy full SHA for aaf19cf

• Fix aiohttp cache

  clnoll

  committed
  5f16581

  Copy full SHA for 5f16581

• Working: handle token-based auth

  clnoll

  committed
  343ba2b

  Copy full SHA for 343ba2b

• Working: refactor BaseHttpStream out of HttpStream & AsyncHttpStream

  clnoll

  committed
  b8675a7

  Copy full SHA for b8675a7

• Working: move session logic from Salesforce into AsyncHttpStream

  clnoll

  committed
  8572328

  Copy full SHA for 8572328

• Working: simplify session logic

  clnoll

  committed
  da1f8b8

  Copy full SHA for da1f8b8

• Manage sessions in the source

  clnoll

  committed
  b84069f

  Copy full SHA for b84069f

• Close sessions as the streams are finished processing

  clnoll

  committed
  db25080

  Copy full SHA for db25080

• Working but awkward: handle session with SourceReader

  clnoll

  committed
  d3e2875

  Copy full SHA for d3e2875

• Working: add back in availability check

  clnoll

  committed
  a5e05f0

  Copy full SHA for a5e05f0

• Working: re-add timer

  clnoll

  committed
  77fcbd6

  Copy full SHA for 77fcbd6

• Working: async stream_slices

  clnoll

  committed
  021fd30

  Copy full SHA for 021fd30

• Working: allow a default concurrency limit to be set

  clnoll

  committed
  c943190

  Copy full SHA for c943190

• Working: don't use adapters.py

  clnoll

  committed
  34a90e3

  Copy full SHA for 34a90e3

• Passing: test_streams_core_async.py

  clnoll

  committed
  5fcbd8d

  Copy full SHA for 5fcbd8d

• Working: cleanup

  clnoll

  committed
  90e61de

  Copy full SHA for 90e61de

• Working: check for streams before starting the reads

  clnoll

  committed
  61cb63a

  Copy full SHA for 61cb63a

• Working: only sync stream if it's in the catalog

  clnoll

  committed
  eed55f6

  Copy full SHA for eed55f6

• Working: move ensure_session onto AsyncStream

  clnoll

  committed
  98e5e0d

  Copy full SHA for 98e5e0d

• Working: handle errors from reader thread

  clnoll

  committed
  de25233

  Copy full SHA for de25233

• Working: some error handling fixes

  clnoll

  committed
  0a0d0a7

  Copy full SHA for 0a0d0a7

• Passing: test_abstract_source_async.py

  clnoll

  committed
  c2b7265

  Copy full SHA for c2b7265

• WIP: test_http_async.py

  clnoll

  committed
  5840f9d

  Copy full SHA for 5840f9d

• Fix backoff handling

  clnoll

  committed
  298bff1

  Copy full SHA for 298bff1

• Fix caching

  clnoll

  committed
  b52c8a1

  Copy full SHA for b52c8a1

• Fix cache test

  clnoll

  committed
  8f685f0

  Copy full SHA for 8f685f0

• All new tests passing except form

  clnoll

  committed
  55445f2

  Copy full SHA for 55445f2

• Remove rate limiter; it can be added back in later

  clnoll

  committed
  fcaa5e1

  Copy full SHA for fcaa5e1

• test_rate_limiting_async.py

  clnoll

  committed
  fd0dd81

  Copy full SHA for fd0dd81

• WIP: test_availability_strategy_async.py

  clnoll

  committed
  ec8f2ed

  Copy full SHA for ec8f2ed

• Passing: test_availability_strategy_async.py

  clnoll

  committed
  7f3f48d

  Copy full SHA for 7f3f48d

• Add availability_strategy_async.py and tests

  clnoll

  committed
  f98f61c

  Copy full SHA for f98f61c

• Reorganize scenario-based test helpers

  clnoll

  committed
  29af3ed

  Copy full SHA for 29af3ed

• Revert an unintended change

  clnoll

  committed
  b5122ff

  Copy full SHA for b5122ff

• Reorganize scenario builder

  clnoll

  committed
  abb30f0

  Copy full SHA for abb30f0

• Remove capsys from scenario-based tests as it's flaky; instead,

  Show description for b47821b
  clnoll

  committed
  b47821b

  Copy full SHA for b47821b

• Add scenario-based tests for asyncio-based concurrency

  clnoll

  committed
  5722338

  Copy full SHA for 5722338

• wip

  clnoll

  committed
  1dda388

  Copy full SHA for 1dda388

• fix tests

  clnoll

  committed
  333d482

  Copy full SHA for 333d482

• Update Salesforce

  clnoll

  committed
  5b4d658

  Copy full SHA for 5b4d658

• Fix Salesforce

  clnoll

  committed
  81b7e7e

  Copy full SHA for 81b7e7e

• Salesforce tests

  clnoll

  committed
  3741cc9

  Copy full SHA for 3741cc9

• Handle check_availability StopAsyncIteration

  clnoll

  committed
  475f33f

  Copy full SHA for 475f33f

• cleanup

  clnoll

  committed
  67fe34e

  Copy full SHA for 67fe34e

• StopIteration in SourceReader.__next__ should not be StopAsyncIteration

  clnoll

  committed
  96d1a78

  Copy full SHA for 96d1a78

• Clean up error handling; CDK passing, Salesforce 2 failing due to exception type not being caught

  clnoll

  committed
  537010d

  Copy full SHA for 537010d

• Fix error handling; all Salesforce & CDK tests & comparison with master pass

  clnoll

  committed
  b8ce8db

  Copy full SHA for b8ce8db

• Reorg into separate async_cdk module

  clnoll

  committed
  145ee23

  Copy full SHA for 145ee23

Commits on Jan 7, 2024

• __init__.py

  clnoll

  committed
  c5f4908

  Copy full SHA for c5f4908

• Use SourceDispatcher to route requests to async source

  clnoll

  committed
  cebca84

  Copy full SHA for cebca84

• Fix types and formatting

  clnoll

  committed
  5874671

  Copy full SHA for 5874671

Commits on Jan 8, 2024

• WIP

  clnoll

  committed
  8e3544c

  Copy full SHA for 8e3544c

• Make unified sync/async error

  clnoll

  committed
  63a379c

  Copy full SHA for 63a379c

Commits on Jan 9, 2024

• Remove some repetetive rate limiting code

  clnoll

  committed
  9c7bd9c

  Copy full SHA for 9c7bd9c

• Salesforce: start splitting into sync vs async

  clnoll

  committed
  bfa7124

  Copy full SHA for bfa7124

• Split out Salesforce async

  clnoll

  committed
  6e93ca9

  Copy full SHA for 6e93ca9

• Salesforce async full refresh & sync incremental

  clnoll

  committed
  d6cfc45

  Copy full SHA for d6cfc45