Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possibility to do a partial reprocessing of a DAG #35

Open
NiltonDuarte opened this issue Nov 3, 2022 · 1 comment
Open

Possibility to do a partial reprocessing of a DAG #35

NiltonDuarte opened this issue Nov 3, 2022 · 1 comment

Comments

@NiltonDuarte
Copy link
Contributor

The problem:
Currently, our examples are heavily dependent on dag_run.start_date as the metadata used for _ingestion_ts
And this add an issue that we can't have a partial reprocessing as the dag_run.start_date will be the start date of the reprocessing run and the framework wont be able to correctly find the parquet files on the currect directory _logical_ts={{ execution_date }}/_ingestion_ts={{ dag_run.start_date }}

@NiltonDuarte
Copy link
Contributor Author

Using the XCom as done in the PR #34 in the bitcoin ingestion example solves this problem but I do not like the solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant