Orchestrator, Executor and Executor Spawners #3
Conversation
There was a problem hiding this comment.
Before going deep into the code, I added some comments based on the first understanding of the README. I am a bit confused regarding the roles and functions of Orchestrator, Executor, Worker and the different Spawners. Maybe starting with a description of each role, their main functions and how they interact with each other in a whole flow might be beneficial.
Waiting for you to apply any required changes before continuing the review.
There was a problem hiding this comment.
I getting closer to an understanding but I still have some issues. Maybe this is because my lack of experience in this area. At least this forces you to create a fool-proof codebase (:
In the README description I find there there's some mix between
- data structures such as
OrchestratorandExecutor - external entities such as
Workerwhich is not a data structure (therefore it shouldn't be described incode-style) - mention of
mini-treeandmini-treeswhich again do not exist in our codebase as objects as I understood - description of how to use it with Swarm, which is something external to our library I guess
- testing
In order to make a clearer understanding I propose to:
- Clearly define what are the goals of
summa-aggregationin pratical terms (i.e. scale merkle sum tree building) - Describe how to do that -> There's an Orchestrator who's role is to split the task of building a Merkle Sum Tree across many workers. Explain that workers are instantiated, in production, as cloud machines. There's no need to make reference to the code here or to code entities yet. Also a very elementary graphical illustration would help here
- All the other comments related to the role of each component such as
Spawner,AggregationMerkleSumTree,OrchestratorandExecutorcan be added directly to the code with in-line comments///or to an example file. - Add the description on how to test the whole flow
As a last point it is still not clear to me the role of the mini_tree_server and mini_tree_generator in the whole context
There was a problem hiding this comment.
I think it would be nice to link to this mini-readme from the main readme so that anyone who's deploying the project can easily find it.
| - published: 4000 | ||
| target: 4000 | ||
| deploy: | ||
| replicas: 2 |
There was a problem hiding this comment.
Why 2 replicas, is this for local testing only?
There was a problem hiding this comment.
Yes, the number of replicas as an example
There was a problem hiding this comment.
Yes, It is used when specified service_info to the CloudSpawner.
There was a problem hiding this comment.
For your information, there is a load-balance layer between node and container in the ingress mode, which is a default network in Docker swarm.
https://docs.docker.com/engine/swarm/ingress/#publish-a-port-for-a-service
I will give you an example in summa-aggregation. Let’s assume two executors are spawned on the Orchestrator with replica parameter at 3. which means the request from the executors may be routed to another container.
This is good for liveness of workers, but I do not know how effective it will be to improve performance when there are more replicas than executor.
|
@sifnoc Can you also attach a comment to why we are returning an empty array in |
I was thinking again about this. Can we simply return the actual |
Firstly, we would need to add |
src/executor/local_spawner.rs
Outdated
| @@ -137,7 +187,7 @@ impl ExecutorSpawner for LocalSpawner { | |||
| } | |||
| } | |||
|
|
|||
| #[cfg(feature = "docker")] | |||
| // #[cfg(feature = "docker")] | |||
There was a problem hiding this comment.
Thanks for point out, I will fix it upcoming last commit for #217 PR on summa-solvency.
Orchestratorcan useExecutorSpawnerto manageExecutorsand the containers connected toExecutors.Here are the review requests:
Spawner?Service Spawner?The
service spawneris not fully implemented yet. but the user can deploymini-tree-generatoras services with givendocker-compose.ymlfile.convert~