feat: add data connectors #478

olevski · 2024-10-23T09:40:26Z

Enables the data connectors to work with the new amalthea.

NOTE: The target branch is not main.

/deploy renku=feat-jupyter-free-sessions amalthea-sessions=main renku-ui=andrea/jupyter-free-build extra-values=amalthea-sessions.deployCrd=false

RenkuBot · 2024-10-23T09:41:09Z

You can access the deployment of this PR at https://renku-ci-ds-478.dev.renku.ch

components/renku_data_services/data_connectors/db.py

Co-authored-by: Samuel Gaist <[email protected]> squashme: resolve package version conflicts

olevski · 2024-10-28T20:34:22Z

NOTE: Saved secrets do not work until this is merged and released: SwissDataScienceCenter/csi-rclone#37

But if you do not save the storage secret and then type it in at session start then things will work as expected.

components/renku_data_services/data_connectors/db.py

leafty

Code looks good.

Question: Is this deployment fully functional? It would be nice to test the sessions with different combinations of data connectors (with saved or unsaved secrets, etc.).

components/renku_data_services/notebooks/api.spec.yaml

components/renku_data_services/notebooks/apispec.py

leafty · 2024-10-30T08:12:57Z

components/renku_data_services/notebooks/blueprints.py

+                # TODO: The UI always does this check if it is acceptable/safe
+                # if csr_id in dcs_secrets and csr.configuration is not None:
+                #     raise errors.ValidationError(
+                #         message=f"Overriding the storage configuration for storage with ID {csr_id} "
+                #         "is not allowed because the storage has an associated saved secret.",
+                #     )


Note: I am not sure about the validity of this comment. What if both access_key_id and secret_access_key are required, but I only saved access_key_id?

Yes I meant to remove this. I will remove it now.

Its gone now.

leafty · 2024-10-30T08:20:31Z

components/renku_data_services/notebooks/util/kubernetes_.py

-    return f"{prefix[:12]}-renku-2-{server_hash[:21]}"
+    # !NOTE: For now we limit the server name to a max of 25 characters.
+    # NOTE: This is 12 + 1 + 12 = 25 characters
+    return f"{prefix[:12]}-{server_hash[:12]}"


Now that we have access to all the data-services information, it would be nice to have a human-readable session name:

{prefix}-{ns_slug}-{project_slug}-{launcher_slug}-{hash}

Also, it would be nicer to try to use Slug.from_name for the prefix part to avoid percent-encoded characters in it.

Yes I have been meaning to do this. And now I am using Slug as you suggested. We actually have Slug.from_user which fits this pretty well. I just still have to do some extra cleaning because k8s is picky about the names.

olevski · 2024-10-30T13:12:46Z

Question: Is this deployment fully functional? It would be nice to test the sessions with different combinations of data connectors (with saved or unsaved secrets, etc.).

The only thing that does not work here is decrypting and using saved secrets for data connectors. It should work though once the fix I submitted for the csi rclone is rolled out on dev (the PR for this is merged and it will be released with renku 0.60.0). Rolling out a parallel version of rclone just for a CI deployment does not really work because both rclones step over one another.

leafty

Trying to use the deployment from the UI, this is what I get from trying to launch a session:

{
    "error": {
        "code": 1422,
        "message": "There are errors in the following fields, name: String should match pattern '^[a-z]([-a-z0-9]*[a-z0-9])?$'"
    }
}

olevski · 2024-10-30T13:28:15Z

Trying to use the deployment from the UI, this is what I get from trying to launch a session

Yes I just saw that. I am fixing it rn.

RenkuBot · 2024-10-30T14:51:16Z

Tearing down the temporary RenkuLab deplyoment for this PR.

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski requested a review from a team as a code owner October 23, 2024 09:40

olevski had a problem deploying to renku-ci-ds-478 October 23, 2024 09:40 — with GitHub Actions Failure

olevski marked this pull request as draft October 23, 2024 09:40

olevski had a problem deploying to renku-ci-ds-478 October 23, 2024 09:43 — with GitHub Actions Failure

olevski temporarily deployed to renku-ci-ds-478 October 23, 2024 13:08 — with GitHub Actions Inactive

olevski temporarily deployed to renku-ci-ds-478 October 23, 2024 13:24 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from 3085cbb to ed2325a Compare October 23, 2024 23:17

olevski had a problem deploying to renku-ci-ds-478 October 23, 2024 23:17 — with GitHub Actions Failure

olevski temporarily deployed to renku-ci-ds-478 October 23, 2024 23:30 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from df31b67 to ccb09b7 Compare October 24, 2024 00:23

olevski temporarily deployed to renku-ci-ds-478 October 24, 2024 00:24 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from ccb09b7 to 3062e07 Compare October 24, 2024 00:49

olevski temporarily deployed to renku-ci-ds-478 October 24, 2024 00:49 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from 3062e07 to 34b33aa Compare October 24, 2024 01:01

olevski temporarily deployed to renku-ci-ds-478 October 24, 2024 01:02 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from 34b33aa to ae494a1 Compare October 24, 2024 01:47

olevski temporarily deployed to renku-ci-ds-478 October 24, 2024 01:47 — with GitHub Actions Inactive

leafty reviewed Oct 24, 2024

View reviewed changes

components/renku_data_services/data_connectors/db.py Outdated Show resolved Hide resolved

olevski had a problem deploying to renku-ci-ds-478 October 27, 2024 01:40 — with GitHub Actions Error

olevski force-pushed the feat-add-data-connectors branch from cf07450 to 1eb8c32 Compare October 27, 2024 01:55

olevski had a problem deploying to renku-ci-ds-478 October 27, 2024 01:55 — with GitHub Actions Failure

olevski temporarily deployed to renku-ci-ds-478 October 27, 2024 02:12 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from 1eb8c32 to 6b27170 Compare October 27, 2024 02:21

olevski temporarily deployed to renku-ci-ds-478 October 27, 2024 02:21 — with GitHub Actions Inactive

olevski force-pushed the feat-add-data-connectors branch from 6b27170 to cbb007d Compare October 27, 2024 04:19

olevski temporarily deployed to renku-ci-ds-478 October 27, 2024 04:20 — with GitHub Actions Inactive

olevski force-pushed the release-amaltheas-migration branch from b2532b4 to 62e197a Compare October 28, 2024 13:01

olevski added 2 commits October 28, 2024 10:25

feat: add new apispec for the new amalthea sessions (#360)

4d55808

feat: run notebooks in data service (#375)

bfbc329

Co-authored-by: Samuel Gaist <[email protected]> squashme: resolve package version conflicts

olevski temporarily deployed to renku-ci-ds-478 October 28, 2024 20:41 — with GitHub Actions Inactive

leafty reviewed Oct 29, 2024

View reviewed changes

components/renku_data_services/data_connectors/db.py Outdated Show resolved Hide resolved

olevski temporarily deployed to renku-ci-ds-478 October 29, 2024 18:07 — with GitHub Actions Inactive

squashme: address comments

5d3333b

olevski force-pushed the feat-add-data-connectors branch from ffb3570 to 5d3333b Compare October 29, 2024 18:25

olevski temporarily deployed to renku-ci-ds-478 October 29, 2024 18:25 — with GitHub Actions Inactive

squashme: add authz check

931b35d

olevski requested a review from leafty October 29, 2024 19:07

olevski temporarily deployed to renku-ci-ds-478 October 29, 2024 19:08 — with GitHub Actions Inactive

leafty approved these changes Oct 30, 2024

View reviewed changes

olevski temporarily deployed to renku-ci-ds-478 October 30, 2024 13:09 — with GitHub Actions Inactive

olevski requested a review from leafty October 30, 2024 13:13

leafty reviewed Oct 30, 2024

View reviewed changes

squashme: address comments

729a64e

olevski force-pushed the feat-add-data-connectors branch from 8545562 to 729a64e Compare October 30, 2024 13:24

olevski deployed to renku-ci-ds-478 October 30, 2024 13:25 — with GitHub Actions View deployment

olevski merged commit 91b8e88 into release-amaltheas-migration Oct 30, 2024
14 checks passed

olevski deleted the feat-add-data-connectors branch October 30, 2024 14:50

olevski added a commit that referenced this pull request Oct 30, 2024

feat: add data connectors (#478)

ea89929

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Oct 31, 2024

feat: add data connectors (#478)

0bf2908

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Nov 1, 2024

feat: add data connectors (#478)

6bb73e0

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Nov 4, 2024

feat: add data connectors (#478)

3db08ac

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Nov 6, 2024

feat: add data connectors (#478)

ba65cae

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Nov 8, 2024

feat: add data connectors (#478)

5f46634

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Nov 11, 2024

feat: add data connectors (#478)

684ab13

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

olevski added a commit that referenced this pull request Nov 11, 2024

feat: add data connectors (#478)

66c9e85

Co-authored-by: Samuel Gaist <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add data connectors #478

feat: add data connectors #478

olevski commented Oct 23, 2024 •

edited

Loading

RenkuBot commented Oct 23, 2024

olevski commented Oct 28, 2024

leafty left a comment

leafty Oct 30, 2024

olevski Oct 30, 2024

olevski Oct 30, 2024

leafty Oct 30, 2024

olevski Oct 30, 2024

olevski commented Oct 30, 2024

leafty left a comment

olevski commented Oct 30, 2024

RenkuBot commented Oct 30, 2024

feat: add data connectors #478

feat: add data connectors #478

Conversation

olevski commented Oct 23, 2024 • edited Loading

RenkuBot commented Oct 23, 2024

olevski commented Oct 28, 2024

leafty left a comment

Choose a reason for hiding this comment

leafty Oct 30, 2024

Choose a reason for hiding this comment

olevski Oct 30, 2024

Choose a reason for hiding this comment

olevski Oct 30, 2024

Choose a reason for hiding this comment

leafty Oct 30, 2024

Choose a reason for hiding this comment

olevski Oct 30, 2024

Choose a reason for hiding this comment

olevski commented Oct 30, 2024

leafty left a comment

Choose a reason for hiding this comment

olevski commented Oct 30, 2024

RenkuBot commented Oct 30, 2024

olevski commented Oct 23, 2024 •

edited

Loading