Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove deleted resources when exporting to sink FHIR store and database during incremental run #588

Closed
chandrashekar-s opened this issue Mar 15, 2023 · 4 comments · Fixed by #1183
Closed

Remove deleted resources when exporting to sink FHIR store and database during incremental run #588

chandrashekar-s opened this issue Mar 15, 2023 · 4 comments · Fixed by #1183
Assignees
@chandrashekar-s
Labels
bug Something isn't working P1:must As issue that definitely needs to be implemented in near future.

Comments

@chandrashekar-s
Copy link
Collaborator

During incremental run, the deleted resources have to be deleted from the sink FHIR store path and the database. Currently, as part of the fix #547 these deleted resources will be removed from the merged parquet files.
@jjtswan
Copy link
Collaborator

jjtswan commented Mar 17, 2023

Can you explain the difference between this and #547 ? Is this bug more about handling the case if the export target is a FHIR store and #547 is about deleting the resource in the case of exporting to parquet file?

@chandrashekar-s
Copy link
Collaborator Author

Yes, this is just an extension for #547 to handle the case where the export target is a FHIR store or another JDBC store like Postgres. Since these cases were not in immediate priority, created this ticket to be handled later on.

@jjtswan
Copy link
Collaborator

jjtswan commented Mar 20, 2023

Got it. Thanks for the explanation.

@jjtswan jjtswan changed the title Remove deleted resources in the sink FHIR store and database during incremental run Remove deleted resources when exporting to sink FHIR store and database during incremental run Mar 27, 2023
@chandrashekar-s chandrashekar-s mentioned this issue Dec 14, 2023
Merged
7 tasks
@bashir2 bashir2 added bug Something isn't working P1:must As issue that definitely needs to be implemented in near future. labels Feb 9, 2024
@bashir2
Copy link
Collaborator

bashir2 commented Feb 9, 2024

Given that our "database as DWH" story is maturing (especially with support for SQL-on-FHIR-v2 views), this bug is more important to be fixed. There is also some renewed interest in syncing to another FHIR store (e.g., #825) which again is relevant to this bug. So changing the priority accordingly.

@chandrashekar-s chandrashekar-s self-assigned this Aug 16, 2024
@chandrashekar-s chandrashekar-s mentioned this issue Sep 9, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chandrashekar-s chandrashekar-s

Labels
bug Something isn't working P1:must As issue that definitely needs to be implemented in near future.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Delete resources in the sink DB and FHIR server chandrashekar-s/fhir-data-pipes
3 participants
@bashir2 @jjtswan @chandrashekar-s