[Merged by Bors] - Fix/custom s3 certificates #247

Maleware · 2023-06-02T09:05:39Z

Description

This PR is supposed to enable TLS for spark-k8s for S3 buckets.
Fixes #194
TODO: add s3 certificate for history server access.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Give feedback

Changes are OpenShift compatible
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Options

Reviewer

Give feedback

Code contains useful comments
(Integration-)Test cases added
Documentation added or updated
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)
Options

Acceptance

Give feedback

Feature Tracker has been updated
Proper release label has been added
Options

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

…name

rust/operator-binary/src/pod_driver_controller.rs

rust/operator-binary/src/spark_k8s_controller.rs

…cted by spark

adwk67 · 2023-06-07T14:47:24Z

Openshift tests:

--- PASS: kuttl (1380.65s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev (88.03s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev (100.78s)
        --- PASS: kuttl/harness/spark-examples_openshift-true_spark-3.4.0-stackable0.0.0-dev (48.32s)
        --- PASS: kuttl/harness/spark-examples_openshift-true_spark-3.3.0-stackable0.0.0-dev (52.71s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev_s3-use-tls-false (109.41s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev_s3-use-tls-false (120.17s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-true_spark-3.4.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (115.74s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-true_spark-3.3.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (124.04s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev (120.29s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev (117.92s)
        --- PASS: kuttl/harness/resources_openshift-true_spark-3.4.0-stackable0.0.0-dev (154.66s)
        --- PASS: kuttl/harness/resources_openshift-true_spark-3.3.0-stackable0.0.0-dev (201.79s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.4.0-stackable0.0.0-dev (230.03s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.3.0-stackable0.0.0-dev (208.98s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev (100.56s)
        --- PASS: kuttl/harness/logging_openshift-true_spark-3.3.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (352.51s)
        --- PASS: kuttl/harness/logging_openshift-true_spark-3.4.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (349.18s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev (89.77s)

adwk67 · 2023-06-07T14:49:36Z

Jenkins tests: 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/88/

adwk67 · 2023-06-20T10:31:54Z

Jenkins run (updated): 🟢 https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/89/

razvan

Some comments

tests/templates/kuttl/spark-history-server/04-prepare-bucket.yaml.j2

tests/templates/kuttl/spark-history-server/03-setup-minio.yaml

tests/templates/kuttl/spark-history-server/helm-bitnami-eventlog-minio-values.yaml.j2

adwk67 · 2023-06-20T12:12:21Z

Openshift tests:

--- PASS: kuttl (1815.94s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-true_spark-3.3.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (122.83s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev (141.16s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev (109.38s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev (92.62s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.3.0-stackable0.0.0-dev_s3-use-tls-true (233.02s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev (95.15s)
        --- PASS: kuttl/harness/logging_openshift-true_spark-3.4.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (324.88s)
        --- PASS: kuttl/harness/spark-examples_openshift-true_spark-3.4.0-stackable0.0.0-dev (50.35s)
        --- PASS: kuttl/harness/logging_openshift-true_spark-3.3.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (334.88s)
        --- PASS: kuttl/harness/spark-examples_openshift-true_spark-3.3.0-stackable0.0.0-dev (47.70s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.4.0-stackable0.0.0-dev_s3-use-tls-false (235.11s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.4.0-stackable0.0.0-dev_s3-use-tls-true (264.63s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev (93.35s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev_s3-use-tls-false (135.54s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev (85.85s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.4.0-stackable0.0.0-dev_s3-use-tls-true (163.62s)
        --- PASS: kuttl/harness/spark-history-server_openshift-true_spark-3.3.0-stackable0.0.0-dev_s3-use-tls-false (261.10s)
        --- PASS: kuttl/harness/resources_openshift-true_spark-3.4.0-stackable0.0.0-dev (141.23s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev_s3-use-tls-true (174.72s)
        --- PASS: kuttl/harness/resources_openshift-true_spark-3.3.0-stackable0.0.0-dev (137.79s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-true_spark-3.4.0-stackable0.0.0-dev_ny-tlc-report-0.1.0 (133.19s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-true_spark-3.3.0-stackable0.0.0-dev_s3-use-tls-false (144.36s)

adwk67 · 2023-06-20T13:03:01Z

bors merge

# Description This PR is supposed to enable TLS for spark-k8s for S3 buckets. Fixes #194 TODO: add s3 certificate for history server access. Co-authored-by: Andrew Kenworthy <[email protected]> Co-authored-by: Razvan-Daniel Mihai <[email protected]>

bors · 2023-06-20T13:08:09Z

Pull request successfully merged into main.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

All tests passed

Maleware added 17 commits May 16, 2023 16:34

Removed check for TLS type

9c66f66

WIP adding trust store function and necessary constants

6912b5e

WIP add some more constants for tls

4ba6416

WIP create key and trustore if TLS is SecretClass

5a57269

WIP add truststore from system truststore

4105bdd

WIP remove comments

5c1486e

WIP move tls to init container and alter functions to respect secret …

ed98af7

…name

WIP clean up

140b952

WIP putting TLS into the init container

cac5b4f

WIP Cleaning up some stuff

b0eb4de

WIP making tls check more robust

b3c3ecd

WIP setting tls directory

ed11f8b

WIP removing some checks

9f556e6

WIP adding s3 to tests

c8b53a8

Added tls test

eddb4f6

Extend test_definition for tls

15f17e7

Cleaning up

e4f3e4e

adwk67 self-requested a review June 2, 2023 09:14

merged main

6d8eeed

github-actions bot reviewed Jun 2, 2023

View reviewed changes

adwk67 added 5 commits June 2, 2023 11:45

fixed linting and clippy warnings

394d033

linting

e883ceb

certificate is added to store via initcontainer but not yet used/dete…

b0c27e8

…cted by spark

corrected minio name to match certificate

cf7703e

test fixes for openshift

f15d9c6

linting fix

f5f500d

adwk67 added 2 commits June 7, 2023 17:14

added s3-tls note to docs

d690e05

updated changelog

edc7520

adwk67 and others added 12 commits June 9, 2023 21:43

wip: history server with custom certificate

741d876

Debug certificate problems

de5a4bf

First successful history test.

8b75e76

More cleanups.

8a6594b

Cleaning up

91394ce

Rust fmt

144cab1

Rust fmt again

884ae33

make extraJavaOpts dependent on tls setting

718020e

use bitnami chart for TLS

1db426a

Regenerate test cert.

66b8112

switch history server test to use bitnami chart for minio

ecec20b

linting fix

849c34b

razvan reviewed Jun 20, 2023

View reviewed changes

adwk67 added 3 commits June 20, 2023 14:43

test tweaks

8271f57

Merge branch 'main' into fix/custom-s3-certificates

96911ea

updated changelog and corrected docs reference

3a28e2b

razvan approved these changes Jun 20, 2023

View reviewed changes

bors bot changed the title ~~Fix/custom s3 certificates~~ [Merged by Bors] - Fix/custom s3 certificates Jun 20, 2023

bors bot closed this Jun 20, 2023

bors bot deleted the fix/custom-s3-certificates branch June 20, 2023 13:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - Fix/custom s3 certificates #247

[Merged by Bors] - Fix/custom s3 certificates #247

Maleware commented Jun 2, 2023 •

edited by adwk67

Loading

Author

Reviewer

Acceptance

adwk67 commented Jun 7, 2023

adwk67 commented Jun 7, 2023 •

edited

Loading

adwk67 commented Jun 20, 2023

razvan left a comment

adwk67 commented Jun 20, 2023

adwk67 commented Jun 20, 2023

bors bot commented Jun 20, 2023

[Merged by Bors] - Fix/custom s3 certificates #247

[Merged by Bors] - Fix/custom s3 certificates #247

Conversation

Maleware commented Jun 2, 2023 • edited by adwk67 Loading

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

adwk67 commented Jun 7, 2023

adwk67 commented Jun 7, 2023 • edited Loading

adwk67 commented Jun 20, 2023

razvan left a comment

Choose a reason for hiding this comment

adwk67 commented Jun 20, 2023

adwk67 commented Jun 20, 2023

bors bot commented Jun 20, 2023

Maleware commented Jun 2, 2023 •

edited by adwk67

Loading

adwk67 commented Jun 7, 2023 •

edited

Loading