Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contract failed rows support #2058

Open
tombaeyens opened this issue Apr 17, 2024 · 1 comment
Open

Contract failed rows support #2058

tombaeyens opened this issue Apr 17, 2024 · 1 comment
Assignees
Labels
contract Data contracts

Comments

@tombaeyens
Copy link
Contributor

tombaeyens commented Apr 17, 2024

TODO:
How does this fit into the broader strategy?
What problem or job are we going after?

HF request.

Work out a contract failed rows proposal. Consider the following approach

A failed rows query can be specified. This failed rows query takes the data source name, database name, schema name from the contract as default and allows users to overwrite these properties.

The failed rows query is executed first and results in a table being created. Similar to (and compatible with) dbt failed rows tests.

After the table is created, the contract verification will only measure the number of failed rows (row count of the failed rows table) as a metric.

Ideally the syntax is very similar for dbt failed rows tests as for soda failed rows checks. Eg

columns:
  - name: stage
    checks: 
      - type: failed_rows
        failed_rows_data_source: (optionally override the contract data source)
        failed_rows_database: (optionally override the contract database)
        failed_rows_schema: (optionally override the contract schema)
        failed_rows_dataset: (required specify the name of the failed rows table)
        failed_rows_query: SELECT ... (optionally specify the name of the failed rows table.  if not specified, soda assumes the table already exists like eg when dbt test already created it)

Internal slite spec link https://sodadata.slite.com/app/docs/0vy9umyD92pSML

@tombaeyens tombaeyens added the contract Data contracts label Apr 17, 2024
@tombaeyens tombaeyens self-assigned this Apr 17, 2024
@tools-soda
Copy link

SAS-3260

@tombaeyens tombaeyens changed the title Add contract support for failed rows query Contract failed rows support Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contract Data contracts
Projects
Status: Goals
Development

No branches or pull requests

2 participants