Info on how to retrieve secrets using google secret manager #1505
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
This was
linked to
issues
Jun 21, 2024
AstrakhantsevaAA
requested changes
Jun 21, 2024
| ```sh | ||
| pip install google-cloud-secret-manager | ||
| ``` | ||
| [Google Docs](https://cloud.google.com/secret-manager/docs/reference/libraries) |
There was a problem hiding this comment.
describe what kind of doc it is.
Suggested change
| [Google Docs](https://cloud.google.com/secret-manager/docs/reference/libraries) | |
| [Google Cloud Documentation: Secret Manager client libraries. ](https://cloud.google.com/secret-manager/docs/reference/libraries) |
| {"api_token": "ghp_Kskdgf98dugjf98ghd...."} | ||
| ``` | ||
|
|
||
| In the script `dlt_with_google_secrets_pipeline.py` you can find an example how to use Google Secrets in `dlt` pipelines. |
There was a problem hiding this comment.
instead of this lone put the code snippet from the bottom
There was a problem hiding this comment.
The bottom code shifted here.
|
|
||
| In the script `dlt_with_google_secrets_pipeline.py` you can find an example how to use Google Secrets in `dlt` pipelines. | ||
|
|
||
| ### Points to Note: |
There was a problem hiding this comment.
it should be at the end
|
|
||
| - **Permissions**: Ensure the service account or user credentials you are using have the necessary permissions to access the Secret Manager and the specific secrets. | ||
| - **Secret Format**: This example assumes that the secret is stored in a JSON string format. If your secret is in a different format, you will need to adjust the parsing method accordingly. | ||
| - **Google Cloud Authentication**: Make sure your environment is authenticated with Google Cloud. This can typically be done by setting credentials in `.dlt/secrets.toml` or setting the `GOOGLE_SECRETS__CREDENTIALS` environment variable to the path of your service account key file or the dict of credentials as a string. |
There was a problem hiding this comment.
add an example of credentials also
[google_secrets.credentials]
"project_id" = "<project_id>"
"private_key" = "-----BEGIN PRIVATE KEY-----\n....\n-----END PRIVATE KEY-----\n"
"client_email" = "....gserviceaccount.com"
| - **Secret Format**: This example assumes that the secret is stored in a JSON string format. If your secret is in a different format, you will need to adjust the parsing method accordingly. | ||
| - **Google Cloud Authentication**: Make sure your environment is authenticated with Google Cloud. This can typically be done by setting credentials in `.dlt/secrets.toml` or setting the `GOOGLE_SECRETS__CREDENTIALS` environment variable to the path of your service account key file or the dict of credentials as a string. | ||
|
|
||
| With this setup, you can effectively retrieve secrets stored in Google Cloud Secret Manager and use them in your `dlt` pipelines as dictionaries. |
There was a problem hiding this comment.
you can delete this sentence at all
|
|
||
| # Fetch the list of repositories from the specified organization using the provided API token. | ||
| # The API token is retrieved from the secret data dictionary. | ||
| data = get_repositories(api_token=secret_data["api_token"], organization="dlt-hub") |
There was a problem hiding this comment.
mentioning function which you don;t provide in a code snippet is not cool. let's make it clearer:
# Retrieve secret data as a dictionary for use in other functions.
secret_data = get_secret_dict("temp-secret")
url = f"https://api.github.com/orgs/dlt-hub/repos"
headers = {
"Authorization": f"token {secret_data["api_token"]}",
"Accept": "application/vnd.github+json",
}
response = requests.get(url, headers=headers)
pipeline = dlt.pipeline(
pipeline_name="quick_start", destination="duckdb", dataset_name="mydata"
)
load_info = pipeline.run(response.json())
print(load_info)
There was a problem hiding this comment.
Thanks, added it to the code.
AstrakhantsevaAA
approved these changes
Jun 21, 2024
AstrakhantsevaAA
deleted the
fix/issue-1411-retrieve-secrets-from-google-secrets
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Added info about