Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up packaging #894

Merged
merged 5 commits into from
Mar 15, 2024
Merged

Speed up packaging #894

merged 5 commits into from
Mar 15, 2024

Conversation

nirvn
Copy link
Member

@nirvn nirvn commented Feb 24, 2024

This PR reworks the packaging job workflow to prevent 3 project loads and instead rely on a single initial load. This can lead to massive speed gains on complex projects and/or projects containing remote sources that are slow.

Requires the following libqfieldsync improvements: opengisch/libqfieldsync#68

@suricactus
Copy link
Collaborator

The decision to load the project 3 times was by design to make each step in a Job workflow independent from the rest. Now they will become tightly coupled and "load project" becomes mandatory. Do we have any specific proof how much do we gain for such an important change that is introduced here?

+1 in putting the logic in a single function, but I would prefer to still call it three times, unless clear evidence we really save 20+ seconds on this.

@nirvn
Copy link
Member Author

nirvn commented Feb 24, 2024

@suricactus , the speed gains are easy to replicate. Just take a QGIS project that loads in 10 seconds, then multiply that by 3 to get the time needlessly spent re-loading it 3 times when we can load it only once.

For one of our clients - the Groupements Forestiers Quebec - , a packaging job takes ~5 minutes, each project load taking 1 minute or so on the cloud server. I can provide job IDs, but I think we already looked at one together yesterday.

@nirvn
Copy link
Member Author

nirvn commented Feb 24, 2024

Avoiding the CPU cost of reloading a project three times is also not negligible, that surely reduces stress on the server infrastructure. Project loading is costly unless you have a single geopackage + an OSM basemap layers (e.g., every time you load the project, you re-create layouts, relationships, 2D and 3D map views, refetch metadata of layers for some data providers, etc.).

Here's a packaging that lasted >4 minutes: https://app.qfield.cloud/a/ggeneau/octobre_2023/jobs?id=91a62569-2a24-454a-b9eb-53ea66292374&job-preview=1

Different project, same ~4 minutes issue: https://app.qfield.cloud/a/superviseur_gfca/Projet_PASCAL_20231208/jobs?id=9b020b46-d561-42d8-aa6f-ea8919868a06&job-preview=1

@suricactus
Copy link
Collaborator

Would be great to monitor back the actual speed improvement of these jobs. I have the gut feeling it will be negligible.

@nirvn
Copy link
Member Author

nirvn commented Feb 28, 2024

@suricactus , PR adjusted to match the content of our discussion yesterday.

@suricactus suricactus merged commit 62ea4fb into master Mar 15, 2024
8 checks passed
@suricactus suricactus deleted the speed_up_packaging branch March 15, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants