Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Source] Improve refresh mecanism #528

Open
bruhnild opened this issue Jan 24, 2024 · 1 comment
Open

[Source] Improve refresh mecanism #528

bruhnild opened this issue Jan 24, 2024 · 1 comment
Labels

Comments

@bruhnild
Copy link
Contributor

bruhnild commented Jan 24, 2024

No description provided.

@bruhnild bruhnild added the admin label Jan 24, 2024
@submarcos
Copy link
Member

Actual system with one queue and beat task that refresh periodic task is problematic.

Every 25min, a task is triggered by beat.

This task scan all source that last ending date / time is greater than defined refresh period.

But, if a task is longer, beat tasks are stacked. When the longer task finished, beat tasks are executed N times and triggered N refresh of all concerned sources

Improve beat refresh_source task to not add a source refresh that is already

If not sufficient, split in two celery queues, one for source refresh, other for beat tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants