sim4life.io - WP6: API

[mguidon]

Description

At sim4life.io, we recognize the importance of providing a comprehensive API that is both flexible and user-friendly. As part of this commitment, we have ensured that our API supports a wide range of features:

• Files: upload/download/delete
• Solvers: start/stop/delete jobs, streaming of job progress
• Studies: create/modify/delete/run
• Users: personal preferences, remaining simulation hours

Via API, users will also be able to define the hardware on which their services run.

This is Sparta!

Preview Give feedback

1. Check #credits > 0 before returning solver outputs from api-server osparc-simcore#5236
   a:apiserver t:enhancement +2
   
2. Check #credits>0 before submitting job #1255
   +0
   

Kobayashi Maru

Preview Give feedback

1. API: support and profile isolve workflow osparc-simcore#4604
   3 of 5
   a:api a:apiserver t:enhancement +3
   

Microhistory

Preview Give feedback

1. API: access to wallets/payment for job submission osparc-simcore#4768
   +0
   
2. release of osparc python client osparc-simcore-clients#50
   0 of 4
   enhancement +1
   
3. Computational Backend: Job that exceeds available resources make Run button hang osparc-simcore#4153
   a:dask-service bug +2
   
4. API: Timestamps are wrong osparc-simcore#4231
   a:apiserver +1
   
5. API: I would like to get hold of the solver log even if I stop a job osparc-simcore#4234
   a:api t:enhancement +2
   
6. api-server: Login per Product, not just user. api key/secret include product_name osparc-simcore#4914
   2 of 2
   a:apiserver a:database +2
   
7. API Job logs: Enable streaming of logs osparc-simcore#4652
   a:apiserver +1
   

Quilmes

Preview Give feedback

1. Public API files resources osparc-simcore#4641
   6 of 6
   e2e t:enhancement +2
   

Baklava

Preview Give feedback

1. Studies API: Add public API entrypoint to clone studies osparc-simcore#4651
   a:apiserver +1
   

Sundae

Preview Give feedback

1. PublicAPI: Allow to list all created jobs (support paging) osparc-simcore#4110
   2 of 2
   a:apiserver t:enhancement +2
   

Watermelon

Preview Give feedback

1. PublicAPI: Allow to delete jobs osparc-simcore#4111
   a:apiserver t:enhancement +2
   
2. API: add delete job #733
   type:enhancement +1
   
3. Complete workflow for public API clients. osparc-simcore#4345
   3 of 3
   t:maintenance +1
   
4. Jobs API custom and client-context metadata osparc-simcore#4313
   a:api t:enhancement +2
   
5. ♻️ apiserver: review and increase test coverage before extending osparc-simcore#3626
   a:apiserver t:maintenance +2
   

[pcrespov]

Update for sprint Pastel de Nata

• Defined openapi specs for new API release:
  • ✨ defines openapi specs for new studies resources osparc-simcore#4250
• Implemented/fixed:
  • API: Timestamps are wrong osparc-simcore#4231
  • API: I would like to get hold of the solver log even if I stop a job osparc-simcore#4234
• Reviewed auto-generation of python-client (ongoing)

[pcrespov]

Update Watermelon

Improving API development& release workflow

Specification-based design starts in openapi specs (OAS) which acts as a contract. Both client and server are built to satisfy that contract. The OAS need to be backwards compatible. The core-part of the client is auto-generated.

The following is a list of actions which have been performed this sprint. Each action is a step towards achieving one/multiple bigger goals which are listed next to the actions.

action	goal
automate generation of python client	enable osparc team to develop and support new features fast support for clients from different languages
automatic testing client vs server	ensure quality (quickly detect bugs)
automatic testing of tutorials/documentation	ensure we have great, user-friendly documentation
add client support for new features on server	expose new functionality to users

Extending osparc public API

• Improved testing:
  • New backwards compatibility tests
  • New mocks of backend stack responses for fast testing
• These are the new/changed entrypoints

1. GET /v0/files/page
2. GET /v0/solvers/page
3. GET /v0/solvers/releases/page
4. GET /v0/solvers/{solver_key}/releases/page
5. GET /v0/solvers/{solver_key}/releases/{version}/ports
6. GET /v0/solvers/{solver_key}/releases/{version}/jobs/page
7. GET /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}/metadata
8. PUT /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}/metadata
9. GET /v0/studies/
10. GET /v0/studies/{study_id}
11. GET /v0/studies/{study_id}/ports
12. GET /v0/studies/{study_id}/jobs
13. POST /v0/studies/{study_id}/jobs
14. GET /v0/studies/{study_id}/jobs/{job_id}
15. DELETE /v0/studies/{study_id}/jobs/{job_id}
16. POST /v0/studies/{study_id}/jobs/{job_id}:start
17. POST /v0/studies/{study_id}/jobs/{job_id}:stop
18. POST /v0/studies/{study_id}/jobs/{job_id}:inspect
19. POST /v0/studies/{study_id}/jobs/{job_id}/outputs
20. POST /v0/studies/{study_id}/jobs/{job_id}/outputs/logfile
21. GET /v0/studies/{study_id}/jobs/{job_id}/metadata
22. PUT /v0/studies/{study_id}/jobs/{job_id}/metadata
23. DELETE /v0/files/{file_id}
24. DELETE /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}

See more details comparing swagger views for:

• current openapi.json
• new openapi-dev.json

Snapshot of progress and next steps

Goal

• full tutorial/s in python that demos all entrypoints above
• online doc

[elisabettai]

See more details comparing swagger views for:

current [openapi.json](https://editor.swagger.io/?url=https://raw.githubusercontent.com/pcrespov/sandbox-python/c1d3e4f7565a26461f510df8f333548cfa83e3ce/api/openapi.json)
new [openapi-dev.json](https://editor.swagger.io/?url=https://raw.githubusercontent.com/pcrespov/sandbox-python/c1d3e4f7565a26461f510df8f333548cfa83e3ce/api/openapi-dev.json)

@pcrespov, am I missing something or these two links above point to the same version of the API? I'm unable to see the diffs by eye in Swagger. 🙃

[pcrespov]

@elisabettai no, one is pointing to the current openapi.json and the other the the new under dev . The diff is the list that I show you above. I also have an automatic report with the diff but it is not easy to visualize (it is for machines!)

[bisgaard-itis]

Update Sundae

• Pagination was added for listing the jobs of a solver (PublicAPI: Allow to list all created jobs (support paging) osparc-simcore#4110)
• Endpoints was added for forwarding the upload links for storage directly to the client. This addresses the 5GB upload limit we have been facing (✨ API server endpoints for performing multipart upload directly to S3 osparc-simcore#4596).
• A tutorial was developed to help guide the development of the study endpoints on the server (📝 Tutorial study functionality osparc-simcore-clients#58).

[bisgaard-itis]

Update Baklava

• Cloning of studies/templates was implemented. This is crucial for being able to produce tutorials and documentation which can be shared with anyone and they can run it. It also enables us to have documentation which consists of jupyter notebooks. That means the documentation is testable. PublicAPI: studies w/ computational services osparc-simcore#4177
• Endpoint was added for deleting files on the server side. Public API files resources osparc-simcore#4641
• Associate a checksum with files on the server. This allows for cheap lookup of files (checksum can be computed locally) and reduced number of uploads (with this in place we can check if a file is already on the server before uploading it). This is still work in progress Public API files resources osparc-simcore#4641
• Job Log streaming : API Job logs: Enable streaming of logs osparc-simcore#4652 🚧
• These are the new entrypoints with respect to the released ones and their status (✅-done, 🚧-in progress):
  • GET /v0/files/page ✅
  • POST /v0/files/{file_id}:abort ✅
  • POST /v0/files/{file_id}:complete ✅
  • POST /v0/files/content ✅
  • DELETE /v0/files/{file_id} ✅
  • GET /v0/solvers/page ✅
  • GET /v0/solvers/releases/page ✅
  • GET /v0/solvers/{solver_key}/releases/page ✅
  • GET /v0/solvers/{solver_key}/releases/{version}/ports ✅
  • GET /v0/solvers/{solver_key}/releases/{version}/jobs/page ✅
  • GET /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}/metadata ✅
  • PATCH /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}/metadata ✅
  • DELETE /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id} ✅
  • GET /v0/studies ✅
  • GET /v0/studies/{study_id} ✅
  • POST /v0/studies/{study_id}:clone ✅
  • GET /v0/studies/{study_id}/ports ✅
  • GET /v0/studies/{study_id}/jobs 🚧
  • POST /v0/studies/{study_id}/jobs 🚧
  • GET /v0/studies/{study_id}/jobs/{job_id} 🚧
  • DELETE /v0/studies/{study_id}/jobs/{job_id} 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}:start 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}:stop 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}:inspect 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}/outputs 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}/outputs/logfile 🚧
  • GET /v0/studies/{study_id}/jobs/{job_id}/metadata 🚧
  • PUT /v0/studies/{study_id}/jobs/{job_id}/metadata 🚧

For details, the openapi specs can be found here

[bisgaard-itis]

Update Quilmes

• API: support and profile isolve workflow osparc-simcore#4604:
  • Added functionality for profiling the api server
• Public API files resources osparc-simcore#4641
  • Added checksum to files -> only upload files if they are not already on the server
  • Only missing item in files section: Limit resources for users
• Almost ready to release next version of client to support solver ports
• These are the new entrypoints with respect to the released ones and their status (✅-done, ⬆️-improved, 🚧-in progress):
  • GET /v0/files/page ✅
  • POST /v0/files/{file_id}:abort ✅
  • POST /v0/files/{file_id}:complete ✅
  • POST /v0/files/content ✅ ⬆️
  • DELETE /v0/files/{file_id} ✅
  • GET /v0/solvers/page ✅
  • GET /v0/solvers/releases/page ✅
  • GET /v0/solvers/{solver_key}/releases/page ✅
  • GET /v0/solvers/{solver_key}/releases/{version}/ports ✅
  • GET /v0/solvers/{solver_key}/releases/{version}/jobs/page ✅
  • GET /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}/metadata ✅
  • PATCH /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id}/metadata ✅
  • DELETE /v0/solvers/{solver_key}/releases/{version}/jobs/{job_id} ✅
  • GET /v0/studies ✅
  • GET /v0/studies/{study_id} ✅
  • POST /v0/studies/{study_id}:clone ✅
  • GET /v0/studies/{study_id}/ports ✅
  • GET /v0/studies/{study_id}/jobs 🚧
  • POST /v0/studies/{study_id}/jobs 🚧
  • GET /v0/studies/{study_id}/jobs/{job_id} 🚧
  • DELETE /v0/studies/{study_id}/jobs/{job_id} 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}:start 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}:stop 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}:inspect 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}/outputs 🚧
  • POST /v0/studies/{study_id}/jobs/{job_id}/outputs/logfile 🚧
  • GET /v0/studies/{study_id}/jobs/{job_id}/metadata 🚧
  • PUT /v0/studies/{study_id}/jobs/{job_id}/metadata 🚧

[bisgaard-itis]

Update [Microhistory]

Done and working

• Wallets/payments for job submission

will be done in 3 weeks

• add products to api server (login) and connect to backend
• solver log streaming

will probably not be available in 3 weeks

• filtering of paginated methods

[pcrespov]

Update 7Peaks

Main implementations are

• API Job logs: Enable streaming of logs osparc-simcore#4652

[bisgaard-itis]

Update Kobayashi Maru

• Several improvements to log streaming.
• During the sprint we encountered a load issue in the API server. In order to monitor this in the future ✨ Add metrics endpoint to api server (⚠️ DEVOPS) osparc-simcore#5193. This should also be extended to other services.

There are no current release blockers.
In order to reduce load on API server (and prepare for metamodelling framework) it is important to tackle ITISFoundation/osparc-simcore#4597 soon.