-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add multi-threading in some capacity #154
Comments
PR #159 is aimed at solving this. |
OK, after much testing, I'm going to put this down as inconclusive. Brief highlights of my testing (which focused on the non-NLP tasks, specifically about 5M of them, generated by synthea
At this point, I had spent so much time on this, I just threw the refactoring I had done into a PR (#160) and left out the actual multi-threading. I think this could be revived in the future, but is not as promising an avenue as I had hoped. Parallelizing the NLP requests though... I think there's value there. But it doesn't have to be threading -- just sending more than one request to cTAKES at a time, probably. Anyway, that's a separate effort from this ticket, I think. I'll make a note of that in the parent performance investigation ticket and close this. |
Right now, Cumulus ETL will only ever use one CPU core. This is criminally ill-performant.
This is a break-out of the larger performance issue #109.
The text was updated successfully, but these errors were encountered: