-
-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up packaging #894
Speed up packaging #894
Conversation
The decision to load the project 3 times was by design to make each step in a Job workflow independent from the rest. Now they will become tightly coupled and "load project" becomes mandatory. Do we have any specific proof how much do we gain for such an important change that is introduced here? +1 in putting the logic in a single function, but I would prefer to still call it three times, unless clear evidence we really save 20+ seconds on this. |
@suricactus , the speed gains are easy to replicate. Just take a QGIS project that loads in 10 seconds, then multiply that by 3 to get the time needlessly spent re-loading it 3 times when we can load it only once. For one of our clients - the Groupements Forestiers Quebec - , a packaging job takes ~5 minutes, each project load taking 1 minute or so on the cloud server. I can provide job IDs, but I think we already looked at one together yesterday. |
Avoiding the CPU cost of reloading a project three times is also not negligible, that surely reduces stress on the server infrastructure. Project loading is costly unless you have a single geopackage + an OSM basemap layers (e.g., every time you load the project, you re-create layouts, relationships, 2D and 3D map views, refetch metadata of layers for some data providers, etc.). Here's a packaging that lasted >4 minutes: https://app.qfield.cloud/a/ggeneau/octobre_2023/jobs?id=91a62569-2a24-454a-b9eb-53ea66292374&job-preview=1 Different project, same ~4 minutes issue: https://app.qfield.cloud/a/superviseur_gfca/Projet_PASCAL_20231208/jobs?id=9b020b46-d561-42d8-aa6f-ea8919868a06&job-preview=1 |
Would be great to monitor back the actual speed improvement of these jobs. I have the gut feeling it will be negligible. |
ecf9668
to
e0f7b5c
Compare
@suricactus , PR adjusted to match the content of our discussion yesterday. |
9ee7233
to
de20d57
Compare
de20d57
to
4d9006e
Compare
This PR reworks the packaging job workflow to prevent 3 project loads and instead rely on a single initial load. This can lead to massive speed gains on complex projects and/or projects containing remote sources that are slow.
Requires the following libqfieldsync improvements: opengisch/libqfieldsync#68