[FEATURE]: Add context info to the output

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Currently we only report warnings and errors. It is unknown when the check was performed and in which pipeline.

Proposed Solution

Add _context column to the output containing: timestamp, workflow it was run on.

Additional Context

No response

[pierre-monnet]

@mwojtyczka
What do you think about replace the current structure of _warning and _error
FROM

{
  col_col1_is_null: "Column col1 is null",
  col_col2_is_null: "Column col2 is null"
}

TO

{
  "col_col1_is_null": {
    "rule": "is_not_null",
    "col": "col1",
    "filter": "col2<3",
    "message": "col1 is not null",
    "execution_date": "2025-01-29 13:37:00",
    "workflow_name": "my_workflow"
  },
  "col_col2_is_null": {
    "rule": "is_not_null",
    "col": "col2",
    "message": "col2 is not null",
    "execution_date": "2025-01-29 13:37:00",
    "workflow_name": "my_workflow"
  }
}

[mwojtyczka]

thanks for the proposal. It looks good with minor changes. I agree, reusing existing reporting columns is more elegant. The only disadvantage is that the execution time will be the same for the same row so potentially repeated info.

I would drop the workflow name or run_id as this info will be available in the Unity Catalog lineage when the dataframe is saved, along with info about the input datasets so we can skip these. Using more contextual info would also require more knowledge about the environment. This could potentially be retrieved but would rather require a separate table.

For the time we should use ISO format. We can use shorter name to save some bytes like "run_time".

{
  "col_col1_is_null": {
    "rule": "is_not_null",
    "col": "col1",
    "filter": "col2<3",
    "message": "col1 is not null",
    "run_time": "2025-01-29T13:37:00",
  },
  "col_col2_is_null": {
    "rule": "is_not_null",
    "col": "col2",
    "message": "col2 is not null",
    "run_time": "2025-01-29T13:37:00",
  }
}

[gergo-databricks]

Question: How would we create this "filter" string? str(DQRule.check)?

I recommend using a timestamp for the time value. Most users of this library will store this data in a table either in this structure or even flatten it, so it's better to store it properly. This would mean converting the current Map to a Struct type.

I propose also giving some flexibility here and creating an option for users to add their metadata.