Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Polling Multiple TaskRuns in One PollTaskRequest #1086

Open
coltmcnealy-lh opened this issue Nov 3, 2024 · 0 comments
Open

Allow Polling Multiple TaskRuns in One PollTaskRequest #1086

coltmcnealy-lh opened this issue Nov 3, 2024 · 0 comments
Labels
clients Affects all LH clients and the public API. May be used in conjunction with `server`. controversial Requires discussion from the community. For proposals, API changes, and big features. feature Issue that denotes a new feature, request, or performance improvement. server Affects the core LH Server code.

Comments

@coltmcnealy-lh
Copy link
Member

@eduwercamacaro did fantastic work on performance testing in LittleHorse. He found out that throughput was bottlenecked by the round trip time of the following cycle in the Task Worker:

  • PollTaskRequest is sent by the client and received by the server.
  • Server sends a TaskClaimEvent to Kafka, then reads it back, and processes it
  • Server returns ScheduledTask to the client.

Right now, only one TaskRun can be in that loop at a time for a single worker thread (PollThread in Java, and connection to the LH Server in Python and Go).

We can improve task worker throughput by parallelizing this. One way is thorugh #1084 which allows multiple loops to go on at once.

Another way is to allow one processing loop (i.e. one PollTaskRequest) to retrieve multiple ScheduledTasks.

Both ways are useful, especially when combined together. The specific benefit of this ticket is that it will reduce the number of network requests and traffic, thereby reducing the load on the server.

We can even take it a step further and allow the TaskClaimEvent internal command to support claiming multiple TaskAttempts at one time, which would further reduce the load we send to Kafka.

@coltmcnealy-lh coltmcnealy-lh added feature Issue that denotes a new feature, request, or performance improvement. server Affects the core LH Server code. controversial Requires discussion from the community. For proposals, API changes, and big features. clients Affects all LH clients and the public API. May be used in conjunction with `server`. labels Nov 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
clients Affects all LH clients and the public API. May be used in conjunction with `server`. controversial Requires discussion from the community. For proposals, API changes, and big features. feature Issue that denotes a new feature, request, or performance improvement. server Affects the core LH Server code.
Projects
None yet
Development

No branches or pull requests

1 participant