Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize feed fetching and score calculation #1

Open
bergie opened this issue Mar 18, 2011 · 2 comments
Open

Parallelize feed fetching and score calculation #1

bergie opened this issue Mar 18, 2011 · 2 comments

Comments

@bergie
Copy link
Contributor

bergie commented Mar 18, 2011

Now we only fetch one feed at a time and import. Feed importing would pass through a lot faster if we had multiple worker processes handling it.

@indeyets
Copy link
Contributor

but proper implementation of this would require some kind of queueing solution. do we have a bug for that?

We need a queue of tasks, with per-task locking and unlocking (if unsuccessful). then we can have multi-process dequeuer, which is started by cron each minute

@bergie
Copy link
Contributor Author

bergie commented Mar 18, 2011

Queue implementation would indeed make this easier. Would you like to propose how it would work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants