This repository has been archived by the owner on Oct 12, 2023. It is now read-only.
Enable OBSClient security provider policy chain #1250
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kiwik
force-pushed
the
enable-obsclient-policy
branch
from
b77f5a1 to
a849f67
Compare
jerrychenhf
reviewed
Mar 23, 2023
| managed_cloud_storage = { | ||
| HWC_MANAGED_CLOUD_STORAGE_OBS_BUCKET: bucket_name, | ||
| CLOUDTIK_MANAGED_CLOUD_STORAGE_URI: managed_cloud_storage_uri} | ||
| CLOUDTIK_MANAGED_CLOUD_STORAGE_URI: _storage_uri} |
There was a problem hiding this comment.
CloudTik (Spark) data access to cloud storage are basically through Hadoop File System. So we originally target the cloud storage URI as Hadoop file system uri to access the bucket. for example:
for AWS S3, its s3a://bucket_name, for GCS: gcs://bucket_name, and so on.
This information is shown as workspace info which is useful for user to access the data in their jobs using this uri.
It's not the URI that in the concept of cloud provider.
For Huawei Cloud, Hadoop file system URI to access may look like: obs://bucket_name or something.
There was a problem hiding this comment.
So that's it, thank you
kiwik
force-pushed
the
enable-obsclient-policy
branch
2 times, most recently
from
8d84c74 to
815f262
Compare
1. Enable OBSClient security provider policy chain, it can work with ENV "OBS_ACCESS_KEY_ID" and "OBS_SECRET_ACCESS_KEY" or ECS agent to get AK/SK automatically, it's disable by default. 2. Fix cloud.storage.uri return a whole OBS bucket URI for command "cloudtik workspace info" 3. Fix ECS create server error for command "cloudtik start", remove unnecessary item in dict of create_server. Related-with: oap-project#1011
kiwik
force-pushed
the
enable-obsclient-policy
branch
from
815f262 to
8986665
Compare
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
with ENV "OBS_ACCESS_KEY_ID" and "OBS_SECRET_ACCESS_KEY" or
ECS agent to get AK/SK automatically, it's disable by default.
command "cloudtik workspace info"
remove unnecessary item in dict of create_server.
Related-with: #1011