refactor(tasks/routes): process uploaded files in parallel as celery groups #504
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
2 times, most recently
from
cd6ab84
to
2d2498b
Compare
matthiasschaub
changed the title
matthiasschaub
changed the title
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
3 times, most recently
from
b59b217
to
93ad756
Compare
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
5 times, most recently
from
4ce2b37
to
e742f2d
Compare
instead of georeferencing uploaded files in one celery tasks sequential (w/ the output of a single big zip file which if too big leads to memory issues) files are now processed in parallel in one celery group. The same is done for digitize sketches: Digitize files in parallel as celery group instead of as single celery task. Other changes: - refactor: move merge and zip_ to helpers module - refactor(celery): use chord and immutable signature. Use chord for executing task after group is finished.
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
from
718f2f3
to
9120d8a
Compare
into test_routes_api_health.py and test_routes_api_status.py
Build user message from changed info and new errors response fields.
Raster and vector results generated by previous version are stored as single AsyncResult in the result store. These need to be retrieved differently as it is now down with GroupResults.
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
from
459d343
to
fbb243f
Compare
Some tests cluttered the Celery queue, which led to timeout errors for next test cases.
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
from
fbb243f
to
3c3a91c
Compare
of `cleanup_map_frames` and `cleanup_blobs`
matthiasschaub
force-pushed
the
split-georeferencing-into-multiple-tasks
branch
from
3c3a91c
to
6ad8135
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of georeferencing uploaded files in one celery tasks sequential
(w/ the output of a single big zip file, when too big leads to memory
issues) files are now processed in parallel in one salary group.
The same is done for digitize sketches: Digitize files in parallel as
celery group instead of as single celery task.
Closes #498
Closes #399
Other changes:
cleanup_map_framesandcleanup_blobsChanged Dependencies
Open Tasks: