You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We want to be able to run this updating job via cron job in kubernetes, but the way we have it set up currently with docker-compose to run a local postrges as a second service complicates that. Rather than trying to use a second k8s job for that, we should just refactor the level-2 branch to do all the initial loading of the parsed data into the same remote aws db we use for the final data. That way we also don't have to rebuild every time from pg_dump. So the new steps would be something like:
upload parsed xml directly into the aws db (maybe under a different schema, though maybe this can just all be in the same)
Batch parse_case . Write to the remote db takes one second per line (maybe this is due to my connection, it might improve if you are running on a data center), compared to a few milliseconds using the local docker instance. This results in processing a file taking 4h59min8s (35691 xml lines).
We want to be able to run this updating job via cron job in kubernetes, but the way we have it set up currently with docker-compose to run a local postrges as a second service complicates that. Rather than trying to use a second k8s job for that, we should just refactor the level-2 branch to do all the initial loading of the parsed data into the same remote aws db we use for the final data. That way we also don't have to rebuild every time from pg_dump. So the new steps would be something like:
The text was updated successfully, but these errors were encountered: