Skip to content

Commit

Permalink
Use buf to generate stubs (#4806)
Browse files Browse the repository at this point in the history 
* use buf

Signed-off-by: Yee Hing Tong <[email protected]>

* Remove java and cpp.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add es and connect-es to buf.gen.yaml and generate files

Signed-off-by: Eduardo Apolinario <[email protected]>

* Generated go code.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add generated python code.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Uncomment openapiv2 definitions.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove go and python swagger-generated code

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove swagger-related code from makefile and scripts

Signed-off-by: Eduardo Apolinario <[email protected]>

* Stop generating code for cpp and java

Signed-off-by: Eduardo Apolinario <[email protected]>

* Stop go code generation in script

Signed-off-by: Eduardo Apolinario <[email protected]>

* Remove docs generated code

Signed-off-by: Eduardo Apolinario <[email protected]>

* Do not generate openapiv2 asset

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

* Update go.mod

Signed-off-by: Eduardo Apolinario <[email protected]>

* Drop os from flyteadmin service

Signed-off-by: Eduardo Apolinario <[email protected]>

* make go-tidy

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix test

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix datacatalog

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint flyteadmin, flyteplugins, and flytepropeller.

Signed-off-by: Eduardo Apolinario <[email protected]>

* fix up datacatalog tests

Signed-off-by: Yee Hing Tong <[email protected]>

* one more test fix

Signed-off-by: Yee Hing Tong <[email protected]>

* fix tests

Signed-off-by: Yee Hing Tong <[email protected]>

* plugin test

Signed-off-by: Yee Hing Tong <[email protected]>

* more test failure, the spacing seems to be different on gh

Signed-off-by: Yee Hing Tong <[email protected]>

* starting admin

Signed-off-by: Yee Hing Tong <[email protected]>

* Fix flyteadmin tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* change propeller compiler test (#4823)

Signed-off-by: Yee Hing Tong <[email protected]>

* Add regex to fix tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Pin versions of typescript buf plugins

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add utility AssertEqualWithSanitizedRegex and fix flyteadmin tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix integration test

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

* More linting

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint again

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add protoc-gen-openapiv2 as a dependence in flyteidl

Signed-off-by: Eduardo Apolinario <[email protected]>

* Serve swagger file using golang's embed.

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix datacatalog tests

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix a few more flyteadmin tests

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
  • Loading branch information
@eapolinario @wild-endeavor
3 people authored Feb 6, 2024
1 parent bfe61f9 commit 0132cba
Show file tree
Hide file tree
Showing 1,854 changed files with 121,366 additions and 1,084,678 deletions.
3 changes: 3 additions & 0 deletions datacatalog/.golangci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,6 @@ linters-settings:
- default
- prefix(github.com/flyteorg)
skip-generated: true
issues:
exclude:
- copylocks
1 change: 1 addition & 0 deletions datacatalog/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ require (
github.com/google/uuid v1.3.1 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
github.com/googleapis/gax-go/v2 v2.7.1 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
Expand Down
4 changes: 4 additions & 0 deletions datacatalog/go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,8 @@ github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
github.com/go-test/deep v1.0.7 h1:/VSMRlnY/JSyqxQUzQLKVMAskpY/NZKFA5j2P+0pP2M=
github.com/go-test/deep v1.0.7/go.mod h1:QV8Hv/iy04NyLBxAdO9njL0iVPN1S4d/A3NVv1V36o8=
github.com/gofrs/uuid v4.2.0+incompatible h1:yyYWMnhkhrKwwr8gAOcOCYxOOscHgDS9yZgBrnJfGa0=
github.com/gofrs/uuid v4.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
Expand Down Expand Up @@ -230,6 +232,8 @@ github.com/googleapis/gax-go/v2 v2.7.1/go.mod h1:4orTrqY6hXxxaUL4LHIPl6lGo8vAE38
github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
Expand Down
16 changes: 8 additions & 8 deletions datacatalog/pkg/manager/impl/artifact_manager_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,23 @@ import (
stdErrors "errors"
"fmt"
"os"
"reflect"
"testing"
"time"
"reflect"

"github.com/golang/protobuf/proto"
"github.com/golang/protobuf/ptypes"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"github.com/flyteorg/flyte/datacatalog/pkg/repositories/transformers"

"github.com/flyteorg/flyte/datacatalog/pkg/common"
"github.com/flyteorg/flyte/datacatalog/pkg/errors"
repoErrors "github.com/flyteorg/flyte/datacatalog/pkg/repositories/errors"
"github.com/flyteorg/flyte/datacatalog/pkg/repositories/mocks"
"github.com/flyteorg/flyte/datacatalog/pkg/repositories/models"
"github.com/flyteorg/flyte/datacatalog/pkg/repositories/transformers"
"github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/core"
"github.com/flyteorg/flyte/flyteidl/gen/pb-go/flyteidl/datacatalog"
"github.com/flyteorg/flyte/flytestdlib/contextutils"
Expand Down Expand Up @@ -237,7 +238,7 @@ func TestCreateArtifact(t *testing.T) {
var value core.Literal
err = datastore.ReadProtobuf(ctx, dataRef, &value)
assert.NoError(t, err)
assert.Equal(t, value, *getTestArtifact().Data[0].Value)
assert.True(t, proto.Equal(&value, getTestArtifact().Data[0].Value))
})

t.Run("Dataset does not exist", func(t *testing.T) {
Expand Down Expand Up @@ -657,7 +658,6 @@ func TestUpdateArtifact(t *testing.T) {
reflect.DeepEqual(artifact.SerializedMetadata, serializedMetadata)
})).Return(nil)


request := &datacatalog.UpdateArtifactRequest{
Dataset: expectedDataset.Id,
QueryHandle: &datacatalog.UpdateArtifactRequest_ArtifactId{
Expand Down Expand Up @@ -692,7 +692,7 @@ func TestUpdateArtifact(t *testing.T) {
var value core.Literal
err = datastore.ReadProtobuf(ctx, dataRef, &value)
assert.NoError(t, err)
assert.Equal(t, value, *getTestStringLiteralWithValue("value11"))
assert.True(t, proto.Equal(&value, getTestStringLiteralWithValue("value11")))

// data2 was not included in update payload, should be removed
dataRef, err = getExpectedDatastoreLocationFromName(ctx, datastore, testStoragePrefix, expectedArtifact, "data2")
Expand All @@ -706,7 +706,7 @@ func TestUpdateArtifact(t *testing.T) {
assert.NoError(t, err)
err = datastore.ReadProtobuf(ctx, dataRef, &value)
assert.NoError(t, err)
assert.Equal(t, value, *getTestStringLiteralWithValue("value3"))
assert.True(t, proto.Equal(&value, getTestStringLiteralWithValue("value3")))
})

t.Run("Update by artifact tag", func(t *testing.T) {
Expand Down Expand Up @@ -786,7 +786,7 @@ func TestUpdateArtifact(t *testing.T) {
var value core.Literal
err = datastore.ReadProtobuf(ctx, dataRef, &value)
assert.NoError(t, err)
assert.Equal(t, value, *getTestStringLiteralWithValue("value11"))
assert.True(t, proto.Equal(&value, getTestStringLiteralWithValue("value11")))

// data2 was not included in update payload, should be removed
dataRef, err = getExpectedDatastoreLocationFromName(ctx, datastore, testStoragePrefix, expectedArtifact, "data2")
Expand All @@ -800,7 +800,7 @@ func TestUpdateArtifact(t *testing.T) {
assert.NoError(t, err)
err = datastore.ReadProtobuf(ctx, dataRef, &value)
assert.NoError(t, err)
assert.Equal(t, value, *getTestStringLiteralWithValue("value3"))
assert.True(t, proto.Equal(&value, getTestStringLiteralWithValue("value3")))
})

t.Run("Artifact not found", func(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion datacatalog/pkg/repositories/errors/postgres.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ package errors
import (
"errors"
"fmt"
"github.com/flyteorg/flyte/flytestdlib/database"
"reflect"

"github.com/jackc/pgconn"
"google.golang.org/grpc/codes"
"gorm.io/gorm"

catalogErrors "github.com/flyteorg/flyte/datacatalog/pkg/errors"
"github.com/flyteorg/flyte/flytestdlib/database"
"github.com/flyteorg/flyte/flytestdlib/logger"
)

Expand Down
3 changes: 2 additions & 1 deletion datacatalog/pkg/repositories/gormimpl/reservation_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"github.com/flyteorg/flyte/datacatalog/pkg/repositories/interfaces"
"github.com/flyteorg/flyte/datacatalog/pkg/repositories/models"
"github.com/flyteorg/flyte/flytestdlib/promutils"
"github.com/flyteorg/flyte/flytestdlib/utils"
)

func TestCreate(t *testing.T) {
Expand Down Expand Up @@ -62,7 +63,7 @@ func TestDeleteFailure(t *testing.T) {

err := reservationRepo.Delete(context.Background(), expectedReservation.ReservationKey, expectedReservation.OwnerID)
assert.Error(t, err)
assert.Equal(t, "missing entity of type Reservation with identifier dataset_id:<project:\"testProject\" name:\"testDataset\" domain:\"testDomain\" version:\"testVersion\" > tag_name:\"testTag\" ", err.Error())
utils.AssertEqualWithSanitizedRegex(t, "missing entity of type Reservation with identifier dataset_id:{project:\"testProject\" name:\"testDataset\" domain:\"testDomain\" version:\"testVersion\"} tag_name:\"testTag\"", err.Error())
}

func TestGet(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion datacatalog/pkg/repositories/initialize.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ package repositories

import (
"context"
"github.com/flyteorg/flyte/flytestdlib/database"
"reflect"

errors2 "github.com/flyteorg/flyte/datacatalog/pkg/repositories/errors"
"github.com/flyteorg/flyte/datacatalog/pkg/runtime"
"github.com/flyteorg/flyte/flytestdlib/database"
"github.com/flyteorg/flyte/flytestdlib/logger"
"github.com/flyteorg/flyte/flytestdlib/promutils"
)
Expand Down
4 changes: 3 additions & 1 deletion flyteadmin/.golangci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,6 @@ linters-settings:
- default
- prefix(github.com/flyteorg)
skip-generated: true

issues:
exclude:
- copylocks
6 changes: 3 additions & 3 deletions flyteadmin/auth/handlers.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
"google.golang.org/grpc/metadata"
"google.golang.org/grpc/peer"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/runtime/protoiface"
"google.golang.org/protobuf/proto"

"github.com/flyteorg/flyte/flyteadmin/auth/interfaces"
"github.com/flyteorg/flyte/flyteadmin/pkg/common"
Expand Down Expand Up @@ -50,7 +50,7 @@ func (e *PreRedirectHookError) Error() string {
type PreRedirectHookFunc func(ctx context.Context, authCtx interfaces.AuthenticationContext, request *http.Request, w http.ResponseWriter) *PreRedirectHookError
type LogoutHookFunc func(ctx context.Context, authCtx interfaces.AuthenticationContext, request *http.Request, w http.ResponseWriter) error
type HTTPRequestToMetadataAnnotator func(ctx context.Context, request *http.Request) metadata.MD
type UserInfoForwardResponseHandler func(ctx context.Context, w http.ResponseWriter, m protoiface.MessageV1) error
type UserInfoForwardResponseHandler func(ctx context.Context, w http.ResponseWriter, m proto.Message) error

type AuthenticatedClientMeta struct {
ClientIds []string
Expand Down Expand Up @@ -496,7 +496,7 @@ func GetLogoutEndpointHandler(ctx context.Context, authCtx interfaces.Authentica
}

func GetUserInfoForwardResponseHandler() UserInfoForwardResponseHandler {
return func(ctx context.Context, w http.ResponseWriter, m protoiface.MessageV1) error {
return func(ctx context.Context, w http.ResponseWriter, m proto.Message) error {
info, ok := m.(*service.UserInfoResponse)
if ok {
if info.AdditionalClaims != nil {
Expand Down
59 changes: 29 additions & 30 deletions flyteadmin/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ module github.com/flyteorg/flyte/flyteadmin
go 1.21

require (
cloud.google.com/go/iam v0.13.0
cloud.google.com/go/storage v1.29.0
cloud.google.com/go/iam v1.1.5
cloud.google.com/go/storage v1.30.1
github.com/NYTimes/gizmo v1.3.6
github.com/Selvatico/go-mocket v1.0.7
github.com/aws/aws-sdk-go v1.44.2
Expand All @@ -20,17 +20,16 @@ require (
github.com/flyteorg/stow v0.3.8
github.com/ghodss/yaml v1.0.0
github.com/go-gormigrate/gormigrate/v2 v2.1.1
github.com/gogo/protobuf v1.3.2
github.com/golang-jwt/jwt/v4 v4.5.0
github.com/golang/glog v1.1.0
github.com/golang/glog v1.1.2
github.com/golang/protobuf v1.5.3
github.com/google/uuid v1.3.1
github.com/googleapis/gax-go/v2 v2.11.0
github.com/google/uuid v1.4.0
github.com/googleapis/gax-go/v2 v2.12.0
github.com/gorilla/handlers v1.5.1
github.com/gorilla/securecookie v1.1.1
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/grpc-ecosystem/grpc-gateway v1.16.0
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.1-0.20210315223345-82c243799c99
github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0
github.com/gtank/cryptopasta v0.0.0-20170601214702-1f550f6f2f69
github.com/jackc/pgconn v1.14.1
github.com/jackc/pgx/v5 v5.4.3
Expand All @@ -49,12 +48,12 @@ require (
github.com/stretchr/testify v1.8.4
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.35.0
go.opentelemetry.io/otel v1.19.0
golang.org/x/oauth2 v0.12.0
golang.org/x/oauth2 v0.15.0
golang.org/x/time v0.3.0
google.golang.org/api v0.126.0
google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc
google.golang.org/grpc v1.56.1
google.golang.org/protobuf v1.31.0
google.golang.org/api v0.149.0
google.golang.org/genproto v0.0.0-20231212172506-995d672761c0
google.golang.org/grpc v1.60.1
google.golang.org/protobuf v1.32.0
gorm.io/driver/mysql v1.4.4
gorm.io/driver/postgres v1.5.3
gorm.io/driver/sqlite v1.5.4
Expand All @@ -73,10 +72,10 @@ require (
)

require (
cloud.google.com/go v0.110.2 // indirect
cloud.google.com/go/compute v1.20.1 // indirect
cloud.google.com/go v0.111.0 // indirect
cloud.google.com/go/compute v1.23.3 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/pubsub v1.30.0 // indirect
cloud.google.com/go/pubsub v1.33.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.3.1 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535 // indirect
Expand Down Expand Up @@ -105,14 +104,15 @@ require (
github.com/go-sql-driver/mysql v1.7.0 // indirect
github.com/go-test/deep v1.0.7 // indirect
github.com/goccy/go-json v0.4.8 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/s2a-go v0.1.4 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/gorilla/websocket v1.4.2 // indirect
github.com/hashicorp/go-uuid v1.0.2 // indirect
github.com/hashicorp/golang-lru v0.5.4 // indirect
Expand Down Expand Up @@ -172,18 +172,17 @@ require (
go.opentelemetry.io/otel/metric v1.19.0 // indirect
go.opentelemetry.io/otel/sdk v1.19.0 // indirect
go.opentelemetry.io/otel/trace v1.19.0 // indirect
golang.org/x/crypto v0.14.0 // indirect
golang.org/x/crypto v0.16.0 // indirect
golang.org/x/exp v0.0.0-20231005195138-3e424a577f31 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sync v0.3.0 // indirect
golang.org/x/sys v0.13.0 // indirect
golang.org/x/term v0.13.0 // indirect
golang.org/x/text v0.13.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/sync v0.4.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/term v0.15.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.13.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240102182953-50ed04b92917 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240102182953-50ed04b92917 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.66.4 // indirect
gopkg.in/jcmturner/aescts.v1 v1.0.1 // indirect
Expand Down
Loading

0 comments on commit 0132cba

Please sign in to comment.