Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sim4life.io - WP4: Computational backend #950

Open
81 of 90 tasks
Tracked by #878
mguidon opened this issue May 9, 2023 · 12 comments
Open
81 of 90 tasks
Tracked by #878

sim4life.io - WP4: Computational backend #950

mguidon opened this issue May 9, 2023 · 12 comments
Assignees
@GitHK @elisabettai @mguidon @sanderegg @matusdrobuliak66
Labels
PO issue Created by Product owners s4l:web sim4life product in osparc.io

Comments

@mguidon
Copy link
Member

mguidon commented May 9, 2023 •
edited by sanderegg
Loading

Description

With the latest version of sim4life.io, we are introducing an improved computational backend that ensures reliable and efficient job scheduling via the computational backend. Moving forward, all solver jobs will be scheduled via these facilities, enabling users to choose the hardware on which their jobs should run and providing the ability to inspect and operate on the job queue (subject to sufficient permissions).

This robust backend will be capable of handling 100s of concurrent jobs, ensuring that even the busiest periods will not cause any disruptions to service.

Furthermore, the backend functionality will also be made available through the API, allowing for integration with external systems (e.g. the sim4life desktop application) and further expanding the possibilities for users.

Tasks
Preview Give feedback
  1. dask-gateway dependency prevents update to SQLAlchemy 2.0 osparc-simcore#4643
    a:director-v2 a:osparc-gateway-server
    sanderegg
  2. Optimize computational scheduler for 100s of jobs osparc-simcore#4530
    sanderegg
  3. Dask backend: Investigate memory requirements of dask-scheduler when running 100s job osparc-simcore#4525
    a:dask-service
    sanderegg
  4. Resource Control: Queue Control #921
    PO issue s4l:web
    sanderegg
  5. Computational jobs fail if the dask-sidecar restarts while they are running #982
    2 of 3
    Feedback type:bug
    sanderegg
  6. 5Gb limit #617
    1 of 2
    bisgaard-itis
  7. Dask backend: Use cpuset for computational services osparc-simcore#3999
    a:dask-service t:enhancement t:maintenance
    sanderegg
  8. Properly set number of GPUs in dask-sidecar osparc-simcore#5094
    a:clusters-keeper a:dask-service a:director-v2
    sanderegg
  9. Make director-v2 scalable/restartable osparc-simcore#4524
    2 of 3
    a:director-v2 t:maintenance
    GitHK sanderegg
  10. Computational clusters: Shut down computational services when credits hit threshold osparc-simcore#5073
    a:clusters-keeper a:director-v2 computational clusters
    sanderegg
  11. Computational clusters: Allow to define cluster concurrency osparc-simcore#5074
    1 of 3
    a:autoscaling a:frontend a:webserver computational clusters t:enhancement
    sanderegg
  12. CLI call to clean and switch off computational cluster #1196
    sanderegg
  13. Computational clusters: User needs visual feedback on the state of the computational clusters (# workers, running state, ...) osparc-simcore#5000
    a:autoscaling a:dask-service computational clusters t:enhancement
    sanderegg
  14. Improve logging parsing of services such as isolve osparc-simcore#5293
    a:apiserver a:dask-service a:director-v2
    sanderegg
  15. Clusters-keeper: if a machine is pending for too long, it should terminate and retry a few times before giving up osparc-simcore#5436
    a:clusters-keeper t:enhancement
    sanderegg
  16. Autoscaling: mark pending instances as failed after they do not join the swarm for too long osparc-simcore#4880
    a:autoscaling t:enhancement
    sanderegg
  17. logs of sim4life jobs not appearing in UI logs osparc-simcore#5336
    a:apiserver a:dask-service a:director-v2
    mguidon
  18. Dask library: Overcome newly introduced bugs in dask 2023.4.x and up osparc-simcore#4526
    a:dask-service bug
    sanderegg

Enchanted Odyssey
Preview Give feedback
  1. External clusters: Provide metrics of all of them clusters for prometheus osparc-simcore#5493
    7 of 7
    a:clusters-keeper a:infra+ops
    mrnicegyu11 sanderegg

Schoggilebe
Preview Give feedback
  1. External clusters: Enable selected AWS regions osparc-simcore#5497
    4 of 4
    a:autoscaling a:clusters-keeper
    YuryHrytsuk sanderegg
  2. External clusters: jobs sent at interval do not get a machine even if there is still room osparc-simcore#5437
    a:autoscaling bug
    sanderegg
  3. create new updated AMIs for autoscaling based on Ansible osparc-simcore#5294
    a:autoscaling a:infra+ops
    sanderegg
  4. Autoscaling: use labels instead of draining nodes osparc-simcore#5339
    a:autoscaling a:director-v2
    sanderegg
  5. Clusters-keeper: WORKERS_EC2_INSTANCES_ALLOWED_TYPE is not passing 16KB limit for EC2 instances osparc-simcore#5290
    2 of 3
    High Priority
    sanderegg
  6. Computational clusters: when the EC2 vCPU quota is reached the clusters does not start processing #1277
    Feedback type:bug
    sanderegg
  7. Autoscaling/Clusters-keeper: label volumes and network interfaces osparc-simcore#5403
    a:autoscaling a:clusters-keeper
    sanderegg

This is Sparta!
Preview Give feedback
  1. ✨Computational backend: add helpful tracing log osparc-simcore#5218
    a:dask-service
    sanderegg
  2. Secure connection to external computational clusters osparc-simcore#4727
    a:dask-service
    sanderegg
  3. Computational clusters: "erred" jobs prevent workers from being shutdown #1218
    type:bug
    sanderegg
  4. Computational clusters: autoscaling unable to scale down worker #1219
    type:bug
    sanderegg
  5. ✨ added CLUSTERS_KEEPER_EC2_INSTANCES_PREFIX env to facilitate testing (⚠️ devops) osparc-simcore#5251
    a:clusters-keeper a:infra+ops
    sanderegg
  6. 🐛🎨Computational autoscaling: allow multi-machining/processing osparc-simcore#5203
    a:autoscaling
    sanderegg
  7. directorv2 fails to reconnect to new scheduler and remains "waiting for cluster" osparc-simcore#5237
    bug
    pcrespov sanderegg
  8. 🎨⚗️Comp backend/bugfix/work stealing (⚠️ devops) osparc-simcore#5261
    a:dask-service e2e
    sanderegg
  9. ✨Dask backend: Enable /metrics endpoint for scraping with prometheus osparc-simcore#5264
    a:dask-service
    sanderegg
  10. Comp backend testing/observations errors from e2e osparc-simcore#5149
    bug
    matusdrobuliak66 sanderegg
  11. 🐛Fix infinite Waiting for cluster when dask-scheduler is restarted osparc-simcore#5252
    a:director-v2
    sanderegg
  12. 🎨 Clusters-keeper: No need to install AWS CLI, it is already in AMI osparc-simcore#5287
    a:clusters-keeper
    sanderegg

Kobayashi Maru
Preview Give feedback
  1. Add tags to EC2 autoscaled machines for easy retrieval (user_id, wallet_id, ....) osparc-simcore#5087
    a:autoscaling a:clusters-keeper
    sanderegg
  2. Ensure alignment between project progress and state #1181
    Feedback type:bug
    bisgaard-itis sanderegg
  3. Dynamic autoscaler can go in infinite creation loop osparc-simcore#5071
    a:autoscaling bug
    sanderegg
  4. ✨🎨 Preferences: Inactivity and Concurrency osparc-simcore#5024
    a:frontend t:enhancement
    ignapas
  5. 🐛Correctly handle file type of on-demand clusters osparc-simcore#5101
    a:director-v2
    sanderegg
  6. 🐛Bugfix/autoscaling does not scale above limit osparc-simcore#5129
    a:autoscaling
    sanderegg
  7. redirect logs of computational clusters to graylog osparc-simcore#5146
    3 of 3
    a:autoscaling a:clusters-keeper
    sanderegg
  8. Storage/handle file overwrite osparc-simcore#5108
    a:storage
    sanderegg
  9. ♻️Maintenance/remove minio-sdk osparc-simcore#5120
    sanderegg
  10. ✨Clusters-keeper: allow for custom boot scripts (⚠️ devops) osparc-simcore#5141
    a:clusters-keeper
    sanderegg
  11. ✨Computational backend: allow to set custom ec2 tags (⚠️ devops) osparc-simcore#5147
    a:autoscaling a:clusters-keeper
    sanderegg
  12. 🐛 Less stringent regex for AWS tag value osparc-simcore#5155
    a:autoscaling a:clusters-keeper t:maintenance
    sanderegg
  13. ✨Monitoring tools: CLI to monitor what is going on osparc-simcore#5162
    a:infra+ops t:maintenance
    sanderegg
  14. 🐛Computational backend: ensure log level is set properly + increased tracing osparc-simcore#5164
    a:clusters-keeper t:maintenance
    sanderegg
  15. ✨Computational backend: timeout pipeline if WAITING_FOR_CLUSTER state takes more than 10 minutes osparc-simcore#5163
    a:director-v2 t:maintenance
    sanderegg
  16. ✨Monitoring tools: show more information about computational clusters osparc-simcore#5165
    a:infra+ops t:maintenance
    sanderegg
  17. 🐛e2e testing: replacement of sleepers using Playwright (preparation for upscaling of ext clusters) osparc-simcore#5167
    a:autoscaling t:maintenance
    sanderegg
  18. 🐛⚗️Computational backend: tasks in UNKNOWN state are sometimes only temporarily in that state osparc-simcore#5195
    a:director-v2
    sanderegg
  19. 🎨Improving computational clusters cli monitoring tool osparc-simcore#5204
    a:infra+ops t:maintenance
    sanderegg
  20. 🐛Clusters-keeper: calling >100 times for the same new cluster now returns cached information osparc-simcore#5201
    a:clusters-keeper a:director-v2
    sanderegg
  21. ✨Docker pull script uses --ignore-pull-failures flag osparc-simcore#5076
    sanderegg

7Peaks
Preview Give feedback
  1. Computational Backend: management and autoscaling of external clusters osparc-simcore#4159
    2 of 2
    a:dask-service a:director-v2 a:frontend a:webserver
    mguidon sanderegg
  2. Fuse pricing units and service resources osparc-simcore#4958
    1 of 1
    a:dask-service a:director-v2 a:dynamic-sidecar a:webserver
    GitHK matusdrobuliak66
    sanderegg
  3. Autoscaling: settings for AMI and Custom boot script shall be linked and made into a list osparc-simcore#4781
    a:autoscaling
    sanderegg
  4. API framework for iSolve #621
    2 of 2
    PO issue
    bisgaard-itis mguidon
  6. e2e tests: simple sleepers (single vs parallel?) #1180
    bisgaard-itis
  7. Autoscaling: creates new machines although the current one should be sufficient osparc-simcore#4999
    High Priority bug
    sanderegg
  8. ✨Computational autoscaling: find out which EC2 type is necessary osparc-simcore#4975
    a:autoscaling
    sanderegg
  9. 🐛Prevent autoscaling from creating workers indefinitely osparc-simcore#5008
    a:autoscaling bug
    sanderegg
  10. ✨Clusters keeper: shutdown all workers on cluster close osparc-simcore#5010
    a:clusters-keeper
    sanderegg
  11. ♻️ Computational backend: set RAM margin to 10% osparc-simcore#5013
    a:director-v2
    sanderegg
  12. ✨Computational backend: Trace parent node osparc-simcore#5042
    a:director-v2
    sanderegg
  13. ✨Dask sidecar: pass job origin and show child computational service logs in workbench osparc-simcore#5054
    a:dask-service
    sanderegg
  14. 🐛Director-v2: fix unit test osparc-simcore#5018
    a:director-v2
    sanderegg
  15. ♻️Autoscaling: Debug logs for issue with scaling up osparc-simcore#5025
    a:autoscaling
    sanderegg
  16. ✨ Re-introduce aws-library osparc-simcore#5031
    a:autoscaling a:aws-library a:clusters-keeper
    sanderegg
  17. 🐛Dynamic autoscaling: not scaling beside 1 machine osparc-simcore#5026
    a:autoscaling bug
    sanderegg
  18. ♻️ Clusters-keeper: renamed ENV variables osparc-simcore#5032
    a:clusters-keeper
    sanderegg
  19. ✨Clusters-keeper: add CLUSTERS_KEEPER_COMPUTATIONAL_BACKEND_DOCKER_IMAGE_TAG ENV osparc-simcore#5066
    a:clusters-keeper
    sanderegg

Microhistory
Preview Give feedback
  1. Before running the service we shall check if enough credits are available to run the service #1034
    GitHK matusdrobuliak66
    sanderegg
  2. 🐛 Autoscaling/Clusters-keeper unable to resolve SSL certificates osparc-simcore#4915
    a:autoscaling a:clusters-keeper
    sanderegg
  3. ✨Director-v2 passes the wallet ID to clusters-keeper osparc-simcore#4930
    a:director-v2
    sanderegg
  4. 🐛 Computational services are not stopped when a project is deleted (Some computational services run forever with machine load) osparc-simcore#3209
    1 of 1
    a:dask-service bug
    sanderegg

Quilmes
Preview Give feedback
  1. Computational backend: state is reported as Pending even though the computation already started and logs are shown osparc-simcore#4517
    a:director-v2 bug
    sanderegg
  2. UI says that a comp. service is running, while it is not true osparc-simcore#4756
    Feedback a:dask-service bug
    sanderegg
  3. Show timestamps in the output log file for computational services #1126
    Feedback type:enhancement
    sanderegg
  4. Computational backend: service shall be able to define how progress is defined in the service osparc-simcore#4376
    a:dask-service bug
    bisgaard-itis sanderegg

Sundae
Preview Give feedback
  1. ✨Computational backend: passing labels and envs 🗃️ osparc-simcore#4429
    a:dask-service changelog:🎨enhancement
    sanderegg
  2. Computational Backend: Job that exceeds available resources make Run button hang osparc-simcore#4153
    a:dask-service bug
    sanderegg
  3. Comp backend: collect metrics and send in RabbitMQ osparc-simcore#4523
    a:director-v2
    matusdrobuliak66 sanderegg

Baklava
Preview Give feedback
  1. Computational Backend: CompRuns validation error osparc-simcore#4637
    a:director-v2 bug
    sanderegg
@mguidon mguidon added PO issue Created by Product owners s4l:web sim4life product in osparc.io labels May 9, 2023
@mguidon mguidon mentioned this issue May 9, 2023
Open
@mguidon mguidon changed the title sim4life.io - Computational backend sim4life.io - WP4: Computational backend May 9, 2023
@pcrespov pcrespov added this to the Pastel de Nata milestone May 12, 2023
@sanderegg
Copy link
Member

sanderegg commented May 14, 2023 •
edited
Loading

Goal for sprint Pastel de Nata

  • progress on AppTeam Std Simulations, ideally run CF use-case
  • refactoring on computational backend, progress on separating PublicAPI calls from webserver load, return solver progress
  • progress on Public API missing entrypoints, and bug fixes
  • if possible progress on personalized resource limits

@sanderegg
Copy link
Member

sanderegg commented May 31, 2023 •
edited
Loading

Update for sprint Pastel de Nata

Done:

Before on prod (05/31/2023) - all instances of webserver are loaded the same because they are coupled through logs/progress/db handling:
image

After on master (05/31/2023) - each instance is less coupled, still waiting for the ongoing PR to completely separate them:
image

@sanderegg sanderegg mentioned this issue Jun 8, 2023
Merged
2 tasks
@pcrespov pcrespov modified the milestones: Pastel de Nata, Watermelon Jun 12, 2023
@mguidon
Copy link
Member Author

mguidon commented Jul 6, 2023 •
edited by sanderegg
Loading

Update Watermelon

Done:

Ongoing:

  • Robustness improvements/refactoring

@GitHK GitHK modified the milestones: Watermelon, Sundae Jul 24, 2023
@sanderegg
Copy link
Member

sanderegg commented Aug 9, 2023 •
edited
Loading

Update Sundae

Done:

  • bugfixes #4153
  • connection of computational backend to resource usage tracking service #4523
  • new clusters keeper service to automatically create computational clusters in AWS #4591

Ongoing:

@matusdrobuliak66 matusdrobuliak66 modified the milestones: Sundae, Baklava Aug 22, 2023
@sanderegg
Copy link
Member

sanderegg commented Sep 6, 2023 •
edited
Loading

Update Baklava

Done:

Ongoing:

@sanderegg
Copy link
Member

sanderegg commented Sep 6, 2023 •
edited
Loading

The below schema shows the overall architecture for the on-demand clusters.
Some important points here are:

  • the computational clusters are created per user/wallet
  • in case of maintenance in simcore, these clusters shall be able to continue running independently

Image

@sanderegg sanderegg modified the milestones: Baklava, the nameless Sep 18, 2023
@sanderegg
Copy link
Member

sanderegg commented Sep 22, 2023 •
edited by bisgaard-itis
Loading

Update Quilmes

Done:

Ongoing:

@sanderegg
Copy link
Member

sanderegg commented Oct 31, 2023 •
edited
Loading

Update Microhistory

Done and working

  • Separate cluster is created for each set of user/wallet combination on demand in Amazon AWS,
  • Cluster is a primary machine (t2.micro), on which a stack containing dask-scheduler, autoscaling, redis, dask-sidecar services is started, dask-sidecar only runs on worker machines,
  • autoscaling service creates 1 worker machine (g4dn.xlarge),
  • Only computational services that use a pricing unit defined as g4dn.xlarge machine can run,
  • computational service uses the all the resources provided by the machine (a bit less than 16Gb/4CPUs)

--> Running computational service should work for one service at a time, provided they are set up to use a g4dn.xlarge machine type, there is no upscaling of the machines so parallel jobs will have to wait in line (if multiple isolve jobs are sent, they will be executed one after the other).

should work in 3 weeks

  • Cluster shall create correct machine based on plan (not just g4dn.xlarge), so potentially better machine fit/performance,
  • identify computational child jobs (for example started from s4l) and show them in UI
  • maybe upscaling of separate cluster (needs discussions on how to do it, it has influence on costs, etc)

should not be available in 3 weeks

  • upscaling?
  • optimisations

@sanderegg
Copy link
Member

sanderegg commented Nov 28, 2023 •
edited
Loading

Update 7peaks

Summary

It is now possible to run computational services on their required AWS instance types. Also child computational job logs show up in the logs of the parent service (e.g. sim4life/jupyterlab starting a computational job).
Upscaling is still not implemented.

Done

Ongoing

  • bugfixing
  • improvements on user feedback (cluster status, number of machines, etc...)

@sanderegg
Copy link
Member

sanderegg commented Jan 7, 2024 •
edited
Loading

Update Kobayashi Maru

Summary

  • bugfixes:
    • handling of on-demand computational clusters (timeouts, reported states)
    • concurrent computing of tasks
  • monitoring & manual interventions:
    • CLI tool to monitor on-demand computational clusters and dynamic service machines
    • partially clear jobs in a specific cluster
    • allow tracing of created machines via tags on EC2 instances

Done ✅

  • various fixes for GPU-based computational services on multi-GPU machines
  • migration of sleepers test to Playwright framework to have more reliable and more flexible E2E testing and compatibility with on-demand computational clusters
  • various fixes regarding invalid state reported by the computational clusters
  • added timeout in case of non responding cluster for more than 10 minutes
  • improvement of response time when retrieving the computational clusters state via Public API
  • new CLI-based monitoring tool to check current state of auto-scaled EC2 instances and their running states

Problematic issues (being worked on) 🚧

Open Features 🚧

@sanderegg sanderegg modified the milestones: Kobayashi Maru, This is Sparta! Jan 11, 2024
@sanderegg
Copy link
Member

sanderegg commented Jan 30, 2024 •
edited
Loading

Update This is Sparta!

Summary

  • Multi-processing is now working as expected (10 machines will take 10 jobs)
  • Added facilities for tracing jobs/logs
  • Secure transmission of data in the computational backend
  • Stability improvements

Done ✅

Ongoing 🚧

Open Features 🚧

@bisgaard-itis bisgaard-itis removed this from the This is Sparta! milestone Feb 19, 2024
@sanderegg
Copy link
Member

Update Schoggilebe

Summary

  • Using Ansible to create AMIs for AWS (reproducibility of machine images) for both dynamic/computational autoscaled machiens
  • Authorize deployment of computational cluster in different AWS regions
  • Improve labeling of machines, network and volumes for better cost management

Done

Ongoing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GitHK GitHK

@elisabettai elisabettai

@mguidon mguidon

@sanderegg sanderegg

@matusdrobuliak66 matusdrobuliak66

Labels
PO issue Created by Product owners s4l:web sim4life product in osparc.io
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@GitHK @mrnicegyu11 @elisabettai @pcrespov @mguidon @sanderegg @YuryHrytsuk @matusdrobuliak66 @bisgaard-itis