Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#6024] feat(iceberg): refactor Iceberg credential code to reuse credential component in Gravitino server #6021

Merged
merged 7 commits into from
Dec 30, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
iceberg rest cache
FANNG1 committed Dec 28, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit c6e43007ee7d30b42546f30f26e7bf6777f16d7f
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@
package org.apache.gravitino.credential;

public class CredentialConstants {
public static final String CREDENTIAL_PROVIDER_TYPE = "credential-provider-type";
@Deprecated public static final String CREDENTIAL_PROVIDER_TYPE = "credential-provider-type";
public static final String CREDENTIAL_PROVIDERS = "credential-providers";
public static final String CREDENTIAL_CACHE_EXPIRE_RATIO = "credential-cache-expire-ratio";
public static final String CREDENTIAL_CACHE_MAX_SIZE = "credential-cache-max-size";
Original file line number Diff line number Diff line change
@@ -51,6 +51,16 @@ public Credential getCredential(String credentialType, CredentialContext context
return credentialCache.getCredential(credentialCacheKey, cacheKey -> doGetCredential(cacheKey));
}

// Get credential with only one credential provider.
public Credential getCredential(CredentialContext context) {
if (credentialProviders.size() == 0) {
throw new RuntimeException("There are not any credential provider for the catalog.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"There is no credential..."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

} else if (credentialProviders.size() > 1) {
throw new RuntimeException("There are multiple credential providers for the catalog.");
}
return getCredential(credentialProviders.keySet().iterator().next(), context);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a more meaningful exception instead of RuntimeException?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

}

@Override
public void close() {
credentialProviders
@@ -67,6 +77,11 @@ public void close() {
e);
}
});
try {
credentialCache.close();
} catch (IOException e) {
LOG.warn("Close credential cache failed, catalog: {}", catalogName, e);
}
}

private Credential doGetCredential(CredentialCacheKey credentialCacheKey) {

This file was deleted.

Original file line number Diff line number Diff line change
@@ -67,6 +67,13 @@ public class CredentialConfig extends Config {
false /* reserved */))
.build();

public static final ConfigEntry<String> CREDENTIAL_PROVIDERS =
new ConfigBuilder(CredentialConstants.CREDENTIAL_PROVIDERS)
.doc("Credential providers, separated by comma.")
.version(ConfigConstants.VERSION_0_8_0)
.stringConf()
.create();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use toSequence() for this configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated


public static final ConfigEntry<Double> CREDENTIAL_CACHE_EXPIRE_RATIO =
new ConfigBuilder(CredentialConstants.CREDENTIAL_CACHE_EXPIRE_RATIO)
.doc(
@@ -91,4 +98,8 @@ public CredentialConfig(Map<String, String> properties) {
super(false);
loadFromMap(properties, k -> true);
}

public CredentialConfig(boolean loadDefaults) {
super(loadDefaults);
}
}
3 changes: 2 additions & 1 deletion dev/docker/iceberg-rest-server/rewrite_config.py
Original file line number Diff line number Diff line change
@@ -22,7 +22,8 @@
"GRAVITINO_IO_IMPL" : "io-impl",
"GRAVITINO_URI" : "uri",
"GRAVITINO_WAREHOUSE" : "warehouse",
"GRAVITINO_CREDENTIAL_PROVIDER_TYPE" : "credential-provider-type",
"GRAVITINO_CREDENTIAL_PROVIDER_TYPE" : "credential-providers",
"GRAVITINO_CREDENTIAL_PROVIDERS" : "credential-providers",
"GRAVITINO_GCS_CREDENTIAL_FILE_PATH" : "gcs-credential-file-path",
"GRAVITINO_S3_ACCESS_KEY" : "s3-access-key-id",
"GRAVITINO_S3_SECRET_KEY" : "s3-secret-access-key",
Loading