Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider preventing ovewriting new data using the completion tracking tables #356

Open
mikix opened this issue Oct 29, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@mikix
Copy link
Contributor

mikix commented Oct 29, 2024

We currently have some code that prevents you from overwriting new data with old data by inspecting the meta.lastUpdated field. But Epic doesn't yet provide that field.

So it would be nice if we had an alternate way to avoid that error.

But now that we have completion tracking, which provides the export timestamp of a group... we could do a similar thing and check that the group & resource combo don't have a newer timestamp already. This requires doing some pre-update queries, which is a new thing for the Delta Lake formatter. But it doesn't seem like a crazy query.

This is a breakout issue from #296

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant