-
Notifications
You must be signed in to change notification settings - Fork 23
Introduce waiters for Intake Runner, Intake and Intake User #3655
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
yago-123
wants to merge
16
commits into
stackitcloud:main
Choose a base branch
from
yago-123:yg/onboard-intake-waiters
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
d9a792c
Introduce waiters for Intake Runner, Intake and Intake User
0c18d79
Adjust timeouts for Intakes
b05b779
Include example using Intake SDK
be755a7
Include root level CHANGELOG
a038985
Include service level CHANGELOG
64dc074
Bump Intake version
3ec4369
Dependencies
f4cec17
Merge branch 'main' into yg/onboard-intake-waiters
yago-123 4b1c13a
Address review: adjust Go version and region value
0a1f021
removed example
ed916d7
Merge branch 'main' into yg/onboard-intake-waiters
devanshcache 2637908
Update services/intake/wait/wait_test.go
devanshcache 40841d8
remove the empty line
0d818c1
use snake case for variables
63ce414
make region as const
65769c3
added intake service example with version v0.2.0
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| module github.com/stackitcloud/stackit-sdk-go/examples/intake | ||
|
|
||
| go 1.21 | ||
|
|
||
| require ( | ||
| github.com/stackitcloud/stackit-sdk-go/core v0.17.3 | ||
| github.com/stackitcloud/stackit-sdk-go/services/intake v0.2.0 | ||
| ) | ||
|
|
||
| require ( | ||
| github.com/golang-jwt/jwt/v5 v5.3.0 // indirect | ||
| github.com/google/uuid v1.6.0 // indirect | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| github.com/golang-jwt/jwt/v5 v5.3.0 h1:pv4AsKCKKZuqlgs5sUmn4x8UlGa0kEVt/puTpKx9vvo= | ||
| github.com/golang-jwt/jwt/v5 v5.3.0/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= | ||
| github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= | ||
| github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= | ||
| github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= | ||
| github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= | ||
| github.com/stackitcloud/stackit-sdk-go/core v0.17.3 h1:GsZGmRRc/3GJLmCUnsZswirr5wfLRrwavbnL/renOqg= | ||
| github.com/stackitcloud/stackit-sdk-go/core v0.17.3/go.mod h1:HBCXJGPgdRulplDzhrmwC+Dak9B/x0nzNtmOpu+1Ahg= | ||
| github.com/stackitcloud/stackit-sdk-go/services/intake v0.2.0 h1:p/zi4VPoCQWk7/2ubi3hxsqiaye41x/Pl3GXYbPkYOY= | ||
| github.com/stackitcloud/stackit-sdk-go/services/intake v0.2.0/go.mod h1:jOArPjNRkwv4487+9ab3dRG+lM09leu5FiRohbQs9Z4= |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "os" | ||
|
|
||
| sdkConfig "github.com/stackitcloud/stackit-sdk-go/core/config" | ||
| "github.com/stackitcloud/stackit-sdk-go/core/utils" | ||
| "github.com/stackitcloud/stackit-sdk-go/services/intake" | ||
| ) | ||
|
|
||
| func main() { | ||
| region := "eu01" // Region where the resources will be created | ||
| projectId := "PROJECT_ID" // Your STACKIT project ID | ||
|
|
||
| dremioCatalogURI := "DREMIO_CATALOG_URI" // E.g., "https://my-dremio-catalog.data-platform.stackit.run/iceberg" | ||
| dremioTokenEndpoint := "DREMIO_TOKEN_ENDPOINT" // E.g., "https://my-dremio.data-platform.stackit.run/oauth/token" | ||
| dremioPAT := "DREMIO_PERSONAL_ACCESS_TOKEN" // Your Dremio Personal Access Token | ||
| catalogWarehouse := "CATALOG_WAREHOUSE" // Catalog warehouse where the data will be ingested | ||
|
|
||
| intakeUserPassword := "s3cuRe_p@ssW0rd_f0r_1ntake!" // A secure password for the new intake user | ||
|
|
||
| ctx := context.Background() | ||
|
|
||
| intakeClient, err := intake.NewAPIClient( | ||
| sdkConfig.WithRegion(region), | ||
| ) | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Creating API client: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
|
|
||
| // Create an Intake Runner | ||
| createRunnerPayload := intake.CreateIntakeRunnerPayload{ | ||
| DisplayName: utils.Ptr("my-example-runner"), | ||
| MaxMessageSizeKiB: utils.Ptr(int64(10)), | ||
| MaxMessagesPerHour: utils.Ptr(int64(1000)), | ||
| } | ||
| createRunnerResp, err := intakeClient.CreateIntakeRunner(ctx, projectId, region).CreateIntakeRunnerPayload(createRunnerPayload).Execute() | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error creating Intake Runner: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| intakeRunnerId := *createRunnerResp.Id | ||
| fmt.Printf("Triggered creation of Intake Runner with ID: %s. Waiting for it to become active...\n", intakeRunnerId) | ||
|
|
||
| // Create an Intake | ||
| dremioAuthType := intake.CatalogAuthType("dremio") // can also be set to "none" if the catalog is not authenticated | ||
| createIntakePayload := intake.CreateIntakePayload{ | ||
| DisplayName: utils.Ptr("my-example-intake"), | ||
| IntakeRunnerId: utils.Ptr(intakeRunnerId), | ||
| Catalog: &intake.IntakeCatalog{ | ||
| Uri: utils.Ptr(dremioCatalogURI), | ||
| Warehouse: utils.Ptr(catalogWarehouse), | ||
| Namespace: utils.Ptr("example_namespace"), | ||
| TableName: utils.Ptr("example_table"), | ||
| Auth: &intake.CatalogAuth{ | ||
| Type: &dremioAuthType, | ||
| Dremio: &intake.DremioAuth{ | ||
| TokenEndpoint: utils.Ptr(dremioTokenEndpoint), | ||
| PersonalAccessToken: utils.Ptr(dremioPAT), | ||
| }, | ||
| }, | ||
| }, | ||
| } | ||
| createIntakeResp, err := intakeClient.CreateIntake(ctx, projectId, region).CreateIntakePayload(createIntakePayload).Execute() | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error creating Intake: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| intakeId := *createIntakeResp.Id | ||
| fmt.Printf("Triggered creation of Intake with ID: %s. Waiting for it to become active...\n", intakeRunnerId) | ||
|
|
||
| createIntakeUserPayload := intake.CreateIntakeUserPayload{ | ||
| DisplayName: utils.Ptr("my-example-user"), | ||
| Password: utils.Ptr(intakeUserPassword), | ||
| } | ||
| createIntakeUserResp, err := intakeClient.CreateIntakeUser(ctx, projectId, region, intakeId).CreateIntakeUserPayload(createIntakeUserPayload).Execute() | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error creating Intake User: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| intakeUserId := *createIntakeUserResp.Id | ||
| fmt.Printf("Created Intake User with ID: %s\n", intakeUserId) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| v0.2.0 | ||
| v0.3.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| package wait | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
| "net/http" | ||
| "time" | ||
|
|
||
| "github.com/stackitcloud/stackit-sdk-go/core/oapierror" | ||
| "github.com/stackitcloud/stackit-sdk-go/core/wait" | ||
| "github.com/stackitcloud/stackit-sdk-go/services/intake" | ||
| ) | ||
|
|
||
| type APIClientInterface interface { | ||
| GetIntakeRunnerExecute(ctx context.Context, projectId, region, intakeRunnerId string) (*intake.IntakeRunnerResponse, error) | ||
| GetIntakeExecute(ctx context.Context, projectId, region, intakeId string) (*intake.IntakeResponse, error) | ||
| GetIntakeUserExecute(ctx context.Context, projectId, region, intakeId, intakeUserId string) (*intake.IntakeUserResponse, error) | ||
| } | ||
|
|
||
| func CreateOrUpdateIntakeRunnerWaitHandler(ctx context.Context, a APIClientInterface, projectId, region, intakeRunnerId string) *wait.AsyncActionHandler[intake.IntakeRunnerResponse] { | ||
| handler := wait.New(func() (waitFinished bool, response *intake.IntakeRunnerResponse, err error) { | ||
| runner, err := a.GetIntakeRunnerExecute(ctx, projectId, region, intakeRunnerId) | ||
| if err != nil { | ||
| return false, nil, err | ||
| } | ||
|
|
||
| if runner == nil { | ||
| return false, nil, fmt.Errorf("API returned a nil response for Intake Runner %s", intakeRunnerId) | ||
| } | ||
|
|
||
| if runner.Id == nil || runner.State == nil { | ||
| return false, nil, fmt.Errorf("could not get ID or State from response for Intake Runner %s", intakeRunnerId) | ||
| } | ||
|
|
||
| if *runner.Id == intakeRunnerId && *runner.State == intake.INTAKERUNNERRESPONSESTATE_ACTIVE { | ||
| return true, runner, nil | ||
| } | ||
|
|
||
| // The API does not have a dedicated failure state for this resource, | ||
| // so we rely on the timeout for cases where it never becomes active. | ||
| return false, nil, nil | ||
| }) | ||
| handler.SetTimeout(15 * time.Minute) | ||
| return handler | ||
| } | ||
|
|
||
| func DeleteIntakeRunnerWaitHandler(ctx context.Context, a APIClientInterface, projectId, region, intakeRunnerId string) *wait.AsyncActionHandler[intake.IntakeRunnerResponse] { | ||
| handler := wait.New(func() (waitFinished bool, response *intake.IntakeRunnerResponse, err error) { | ||
| _, err = a.GetIntakeRunnerExecute(ctx, projectId, region, intakeRunnerId) | ||
| if err == nil { | ||
| // Resource still exists | ||
| return false, nil, nil | ||
| } | ||
|
|
||
| var oapiError *oapierror.GenericOpenAPIError | ||
| if errors.As(err, &oapiError) { | ||
| if oapiError.StatusCode == http.StatusNotFound { | ||
| // Success: Resource is gone | ||
| return true, nil, nil | ||
| } | ||
| } | ||
| // An unexpected error occurred | ||
| return false, nil, err | ||
| }) | ||
| handler.SetTimeout(15 * time.Minute) | ||
| return handler | ||
| } | ||
|
|
||
| func CreateOrUpdateIntakeWaitHandler(ctx context.Context, a APIClientInterface, projectId, region, intakeId string) *wait.AsyncActionHandler[intake.IntakeResponse] { | ||
| handler := wait.New(func() (waitFinished bool, response *intake.IntakeResponse, err error) { | ||
| ik, err := a.GetIntakeExecute(ctx, projectId, region, intakeId) | ||
| if err != nil { | ||
| return false, nil, err | ||
| } | ||
|
|
||
| if ik == nil { | ||
| return false, nil, fmt.Errorf("API returned a nil response for Intake %s", intakeId) | ||
| } | ||
|
|
||
| if ik.Id == nil || ik.State == nil { | ||
| return false, nil, fmt.Errorf("could not get ID or State from response for Intake %s", intakeId) | ||
| } | ||
|
|
||
| if *ik.Id == intakeId && *ik.State == intake.INTAKERESPONSESTATE_ACTIVE { | ||
| return true, ik, nil | ||
| } | ||
|
|
||
| if *ik.Id == intakeId && *ik.State == intake.INTAKERESPONSESTATE_FAILED { | ||
| return true, ik, fmt.Errorf("create/update failed for Intake %s", intakeId) | ||
| } | ||
|
|
||
| return false, nil, nil | ||
| }) | ||
| handler.SetTimeout(10 * time.Minute) | ||
| return handler | ||
| } | ||
|
|
||
| func DeleteIntakeWaitHandler(ctx context.Context, a APIClientInterface, projectId, region, intakeId string) *wait.AsyncActionHandler[intake.IntakeResponse] { | ||
| handler := wait.New(func() (waitFinished bool, response *intake.IntakeResponse, err error) { | ||
| _, err = a.GetIntakeExecute(ctx, projectId, region, intakeId) | ||
| if err == nil { | ||
| return false, nil, nil | ||
| } | ||
|
|
||
| var oapiError *oapierror.GenericOpenAPIError | ||
| if errors.As(err, &oapiError) { | ||
| if oapiError.StatusCode == http.StatusNotFound { | ||
| return true, nil, nil | ||
| } | ||
| } | ||
| return false, nil, err | ||
| }) | ||
| handler.SetTimeout(10 * time.Minute) | ||
| return handler | ||
| } | ||
|
|
||
| func CreateOrUpdateIntakeUserWaitHandler(ctx context.Context, a APIClientInterface, projectId, region, intakeId, intakeUserId string) *wait.AsyncActionHandler[intake.IntakeUserResponse] { | ||
| handler := wait.New(func() (waitFinished bool, response *intake.IntakeUserResponse, err error) { | ||
| user, err := a.GetIntakeUserExecute(ctx, projectId, region, intakeId, intakeUserId) | ||
| if err != nil { | ||
| return false, nil, err | ||
| } | ||
|
|
||
| if user == nil { | ||
| return false, nil, fmt.Errorf("API returned a nil response for Intake User %s", intakeUserId) | ||
| } | ||
|
|
||
| if user.Id == nil || user.State == nil { | ||
| return false, nil, fmt.Errorf("could not get ID or State from response for Intake User %s", intakeUserId) | ||
| } | ||
|
|
||
| if *user.Id == intakeUserId && *user.State == intake.INTAKEUSERRESPONSESTATE_ACTIVE { | ||
| return true, user, nil | ||
| } | ||
|
|
||
| // The API does not have a dedicated failure state for this resource, we rely on the timeout for cases where | ||
| // it never becomes active. | ||
| return false, nil, nil | ||
| }) | ||
| handler.SetTimeout(5 * time.Minute) | ||
| return handler | ||
| } | ||
|
|
||
| func DeleteIntakeUserWaitHandler(ctx context.Context, a APIClientInterface, projectId, region, intakeId, intakeUserId string) *wait.AsyncActionHandler[intake.IntakeUserResponse] { | ||
| handler := wait.New(func() (waitFinished bool, response *intake.IntakeUserResponse, err error) { | ||
| _, err = a.GetIntakeUserExecute(ctx, projectId, region, intakeId, intakeUserId) | ||
| if err == nil { | ||
| return false, nil, nil | ||
| } | ||
|
|
||
| var oapiError *oapierror.GenericOpenAPIError | ||
| if errors.As(err, &oapiError) { | ||
| if oapiError.StatusCode == http.StatusNotFound { | ||
| return true, nil, nil | ||
| } | ||
| } | ||
| return false, nil, err | ||
| }) | ||
| handler.SetTimeout(5 * time.Minute) | ||
| return handler | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In one of your last commits, you removed the examples again. Is there a reason why they are removed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Linter failed because the examples referenced the
v0.3.0intake SDK module which isn't available yet ...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yago-123 wrote me he will create a follow-up PR with the examples
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright. But then the examples won't be part of this release, so the changelogs should be adjusted
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yago-123
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its fine by us if its in next release