Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microbatch: event_time ref + source filtering #10594

Merged
merged 41 commits into from
Sep 12, 2024
Merged

Microbatch: event_time ref + source filtering #10594

merged 41 commits into from
Sep 12, 2024

Commits on Aug 22, 2024

  1. initial rough-in with CLI flags

    @MichelleArk
    MichelleArk committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    c4930e9 View commit details
    Browse the repository at this point in the history

  2. dbt-adapters testing against event-time-ref-filtering

    @MichelleArk
    MichelleArk committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4c8528b View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Merge branch 'main' into event-time-ref-filtering

    @MichelleArk
    MichelleArk committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    3bb6807 View commit details
    Browse the repository at this point in the history

  2. fix TestList

    @MichelleArk
    MichelleArk committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    f5d5bb6 View commit details
    Browse the repository at this point in the history

  3. Checkpoint

    @QMalcolm
    QMalcolm committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    19ad7c6 View commit details
    Browse the repository at this point in the history

  4. fix tests

    @MichelleArk
    MichelleArk committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    a57481f View commit details
    Browse the repository at this point in the history

  5. add event_time_start params to build

    @MichelleArk
    MichelleArk committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    25c10f7 View commit details
    Browse the repository at this point in the history

  6. rename configs

    @MichelleArk
    MichelleArk committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    699179f View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Gate resolve_event_time_filter via micro batch strategy and fix strpt… 

    …ime usage
    @QMalcolm
    QMalcolm committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    2d19d1c View commit details
    Browse the repository at this point in the history

  2. Add unit test for resolve_event_time_filter

    @QMalcolm
    QMalcolm committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    57b1353 View commit details
    Browse the repository at this point in the history

  3. Additional unit tests for resolve_event_time_filter to ensure lookb… 

    …ack + batch_size work
    @QMalcolm
    QMalcolm committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    e0bae27 View commit details
    Browse the repository at this point in the history

  4. Remove extraneous comments and print statements from resolve_event_ti… 

    …me_filter
    @QMalcolm
    QMalcolm committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    1313aff View commit details
    Browse the repository at this point in the history

  5. Fixup microbatch functional tests to use microbatch strategy

    @QMalcolm
    QMalcolm committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    7307c02 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Gate microbatch functionality behind env_var while in beta

    @QMalcolm
    QMalcolm committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    838a0aa View commit details
    Browse the repository at this point in the history

  2. Add comment about how _is_incremental should be removed

    @QMalcolm
    QMalcolm committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    43715de View commit details
    Browse the repository at this point in the history

  3. Improve event_time_start/end cli parameters to auto convert to date… 

    …time objects
    @QMalcolm
    QMalcolm committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    e38ff47 View commit details
    Browse the repository at this point in the history

  4. for testing: dbt-postgres 'microbatch' strategy

    @MichelleArk
    MichelleArk committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    457698c View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Make event_time model configs Any so as to not break people already… 

    … using those names
    @QMalcolm
    QMalcolm committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    3a4fa7f View commit details
    Browse the repository at this point in the history

  2. Rename PartitionGrain to BatchSize

    @QMalcolm
    QMalcolm committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    8cc1c7d View commit details
    Browse the repository at this point in the history

  3. fix error message about batch size

    @QMalcolm
    QMalcolm committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    1b0a904 View commit details
    Browse the repository at this point in the history

  4. Create unit test to check values which gate off event_time usage

    @QMalcolm
    QMalcolm committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    7522b9b View commit details
    Browse the repository at this point in the history

  5. Add unit test case in regard to event_tiem filtering when `is_increme… 

    …ntal` is false
    @QMalcolm
    QMalcolm committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    5f72534 View commit details
    Browse the repository at this point in the history

  6. Cleanup unit test for test_resolve_event_time_filter_batch_calculation

    @QMalcolm
    QMalcolm committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    a8b9d64 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Merge branch 'main' into event-time-ref-filtering

    @QMalcolm
    QMalcolm committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    174db0a View commit details
    Browse the repository at this point in the history

  2. Update lower bound of dbt-adapters req to 1.5.0 

    Previously our lower bound was 1.3.0. This will no longer work. In dbt-adapters
1.5.0 we released the EventTimeFilter class, and dbt-core now depends on that.
Thus dbt-adapters < 1.5.0 are no longer compatible.
    @QMalcolm
    QMalcolm committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    b85d50e View commit details
    Browse the repository at this point in the history

  3. Update dev-reqs now that the changes we need have been merged to main… 

    … in dbt-adapters
    @QMalcolm
    QMalcolm committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    ec01129 View commit details
    Browse the repository at this point in the history

  4. Remove is_incremental check from _build_end_time 

    Even when `is_incremental` is false, we still plan to _microbatch_
incremental microbatch models. To do so we need to have an end time.
Additionally, some tests were wonky because of the previous behavior
    @QMalcolm
    QMalcolm committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8f8e7e3 View commit details
    Browse the repository at this point in the history

  5. Improve help messages for event_time_start/end CLI flags

    @QMalcolm
    QMalcolm committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    2b80f75 View commit details
    Browse the repository at this point in the history

  6. Improve error handling messages for event time filtering generation

    @QMalcolm
    QMalcolm committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    0eb8e2f View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Merge branch 'main' into event-time-ref-filtering

    @QMalcolm
    QMalcolm committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    3a6c739 View commit details
    Browse the repository at this point in the history

  2. Delete pairing.md file

    @QMalcolm
    QMalcolm committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    1690d81 View commit details
    Browse the repository at this point in the history

  3. Add changie doc for initial microbatch implementation

    @QMalcolm
    QMalcolm committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    de87286 View commit details
    Browse the repository at this point in the history

  4. Update v12.json manifest schema for NodeConfig changes

    @QMalcolm
    QMalcolm committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    12bfec9 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Add event_time to SourceConfig and allow event time filtering for… 

    … sources
    @QMalcolm
    QMalcolm committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    9b47966 View commit details
    Browse the repository at this point in the history

  2. Add test that asserts direct upstrean inputs without event_time aren'… 

    …t filtered
    @QMalcolm
    QMalcolm committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    6767948 View commit details
    Browse the repository at this point in the history

  3. Add test asserting that calling .render() on a ref skips event time… 

    … filtering
    @QMalcolm
    QMalcolm committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    61123cb View commit details
    Browse the repository at this point in the history

  4. Update v12 manifest schema with new SourceConfig key for event_time

    @QMalcolm
    QMalcolm committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    43866db View commit details
    Browse the repository at this point in the history

  5. fix test_list for source config event_time

    @MichelleArk
    MichelleArk committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    5426e66 View commit details
    Browse the repository at this point in the history

  6. restore dbt-postgres dev requirement

    @MichelleArk
    MichelleArk committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ee29fed View commit details
    Browse the repository at this point in the history

  7. Update dbt-adapters minimum version to 1.6.0

    @QMalcolm
    QMalcolm committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    3020aa4 View commit details
    Browse the repository at this point in the history

  8. fix test_artifacts; add event_time to expected_manifest

    @MichelleArk
    MichelleArk committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    f53204f View commit details
    Browse the repository at this point in the history