Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Artifacts #4077

Closed
wants to merge 68 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
34666a2
use-personal
wild-endeavor Aug 31, 2023
efd28b9
tidy
wild-endeavor Aug 31, 2023
165bcf2
update port to 30004
wild-endeavor Sep 4, 2023
df3b046
Artifacts bundle (#7)
wild-endeavor Sep 26, 2023
5d54379
Remove private repos (#4078)
wild-endeavor Sep 26, 2023
f33132a
nit from monorepo merge
wild-endeavor Sep 29, 2023
c9ce299
Artifacts bring in idl (#4204)
wild-endeavor Oct 11, 2023
6792f4e
Artifacts bring admin monorepo (#4203)
wild-endeavor Oct 30, 2023
ee03489
Artf/triggers (#4394)
wild-endeavor Nov 10, 2023
9741afc
make trigger key index unique (#4410)
wild-endeavor Nov 13, 2023
e7d66ec
revert data proxy changes (#4411)
wild-endeavor Nov 13, 2023
568127e
merge master and fix up some unit tests
wild-endeavor Nov 13, 2023
096b7c1
artf/ fix issue with trigger (#4412)
wild-endeavor Nov 13, 2023
05b4ea0
sandbox lite dockerfile (#4413)
wild-endeavor Nov 13, 2023
63b577e
More pgconn handling fixes (#4420)
wild-endeavor Nov 14, 2023
484e144
Artf/switch event (#4428)
wild-endeavor Nov 15, 2023
2e256d3
Artf/lints (#4429)
wild-endeavor Nov 15, 2023
d434103
look for other places where pgconn is cast and add handling (#4430)
wild-endeavor Nov 15, 2023
9584959
add artifacts branch to buf push (#4448)
squiishyy Nov 17, 2023
ba21a65
add dummy comment (#4451)
squiishyy Nov 17, 2023
4dbcba3
add dummy comment again (#4453)
squiishyy Nov 17, 2023
c3ce7aa
artf/updates to source (#4443)
wild-endeavor Nov 21, 2023
fdd8fe9
Merge remote-tracking branch 'origin/master' into artifacts
wild-endeavor Nov 22, 2023
8a895b9
add a feature gate (#4472)
wild-endeavor Nov 23, 2023
9d7e55e
fix bug in execution_manager with lpExpectedInputs, remove the standa…
wild-endeavor Nov 23, 2023
72f8b31
initial deletion of artifact code
wild-endeavor Nov 23, 2023
232a1c7
add back all files according to 9d7e55e9f
wild-endeavor Nov 23, 2023
6cbd980
running make manifests in docker/sandbox-bundled to update generated …
wild-endeavor Nov 24, 2023
f6ea5d9
now compiles (#4481)
wild-endeavor Nov 27, 2023
098e5be
add a debug line
wild-endeavor Dec 1, 2023
ae30de8
more debugging
wild-endeavor Dec 1, 2023
8d8ce67
clean up go mod and switch the order of the input extraction to check…
wild-endeavor Dec 1, 2023
3897e86
change artifacts http mappings in protos (#4554)
squiishyy Dec 8, 2023
e713888
change artifacts http mappings again (#4562)
squiishyy Dec 8, 2023
7a9d6a8
resolve merge conflicts
wild-endeavor Dec 20, 2023
0d38bdc
change artifacts http mappings in protos
squiishyy Dec 7, 2023
1330343
testing removal of go mod pin
wild-endeavor Dec 20, 2023
424db05
update go mod some more
wild-endeavor Dec 20, 2023
fecb836
merge master again
wild-endeavor Dec 20, 2023
c3cfa6f
copilot go mod
wild-endeavor Dec 20, 2023
5771394
change artifacts http mappings again (#4562)
squiishyy Dec 8, 2023
e743da7
panic and revert gorm log level change
wild-endeavor Dec 21, 2023
80267bd
revert local to 5
wild-endeavor Dec 21, 2023
cdae872
Merge branch 'artifacts-shell' of github.com:flyteorg/flyte into arti…
wild-endeavor Dec 21, 2023
a303ec9
make helm
wild-endeavor Dec 21, 2023
6777cb2
Merge remote-tracking branch 'origin/master' into artifacts-shell
wild-endeavor Dec 21, 2023
00545fd
Reclaim space prior to running functional tests in single-binary
eapolinario Dec 21, 2023
cfa4745
Pin flytectl version to 0.8.0
eapolinario Dec 21, 2023
e3cd981
Merge remote-tracking branch 'origin' into artifacts-shell
eapolinario Dec 21, 2023
673a4c1
Revert "Pin flytectl version to 0.8.0"
eapolinario Dec 21, 2023
3528c2e
Use flytagent v1.10.2
eapolinario Dec 21, 2023
7972fca
Debug single-binary
eapolinario Dec 22, 2023
890a80e
Use an infinite loop instead of watch (term env var is not set)
eapolinario Dec 22, 2023
a11d90e
Get logs from single-binary pod
eapolinario Dec 22, 2023
d14a140
Debug single-binary pod: use a different way of choosing the pod
eapolinario Dec 22, 2023
3968d40
Fix flytectl version
eapolinario Dec 22, 2023
8ded67b
Use port 5432 to connect to postgres
eapolinario Dec 22, 2023
3c7dbf6
Revert "Use port 5432 to connect to postgres"
eapolinario Dec 22, 2023
dd49bb2
Revert "Fix flytectl version"
eapolinario Dec 22, 2023
8ce70c2
fix database error
wild-endeavor Dec 23, 2023
7872748
Merge remote-tracking branch 'origin/master' into artifacts-shell
wild-endeavor Dec 23, 2023
ce5cb7b
revert .github/workflows/single-binary.yml by pulling from master
wild-endeavor Dec 23, 2023
8e8f1cf
Merge branch 'artifacts-shell' into artifacts
wild-endeavor Dec 26, 2023
f60ae81
merge master and making compile
wild-endeavor Dec 27, 2023
c06527e
Artf/cleanup post shell (#4646)
wild-endeavor Dec 29, 2023
7211dbb
artf/make goimports (#4651)
wild-endeavor Dec 29, 2023
7557d62
merge master and regenerate docker sandbox-bundled manifests
wild-endeavor Jan 8, 2024
de919c0
Artf/lineage and fixes (#4680)
wild-endeavor Jan 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ jobs:
component:
- datacatalog
- flyteadmin
- flyteartifacts
# TODO(monorepo): Enable lint flytecopilot
# - flytecopilot
- flyteidl
Expand Down Expand Up @@ -69,6 +70,7 @@ jobs:
component:
- datacatalog
- flyteadmin
- flyteartifacts
- flytecopilot
- flytepropeller
name: Docker Build Images
Expand Down Expand Up @@ -112,6 +114,7 @@ jobs:
component:
- datacatalog
- flyteadmin
- flyteartifacts
- flytecopilot
- flytepropeller
uses: ./.github/workflows/go_generate.yml
Expand Down
1 change: 1 addition & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ WORKDIR /flyteorg/build

COPY datacatalog datacatalog
COPY flyteadmin flyteadmin
COPY flyteartifacts flyteartifacts
COPY flytecopilot flytecopilot
COPY flyteidl flyteidl
COPY flyteplugins flyteplugins
Expand Down
42 changes: 42 additions & 0 deletions Dockerfile.flyteartifacts
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
FROM --platform=${BUILDPLATFORM} golang:1.21.5-alpine3.18 as builder

ARG TARGETARCH
ENV GOARCH "${TARGETARCH}"
ENV GOOS linux

RUN apk add git openssh-client make curl

# Create the artifacts directory
RUN mkdir /artifacts

WORKDIR /go/src/github.com/flyteorg/flyte/flyteartifacts/

COPY datacatalog ../datacatalog
COPY flyteadmin ../flyteadmin
COPY flyteartifacts .
COPY flytecopilot ../flytecopilot
COPY flyteidl ../flyteidl
COPY flyteplugins ../flyteplugins
COPY flytepropeller ../flytepropeller
COPY flytestdlib ../flytestdlib

# This 'linux_compile' target should compile binaries to the /artifacts directory
# The main entrypoint should be compiled to /artifacts/flyteartifacts
RUN make linux_compile

# update the PATH to include the /artifacts directory
ENV PATH="/artifacts:${PATH}"

# This will eventually move to centurylink/ca-certs:latest for minimum possible image size
FROM alpine:3.16
LABEL org.opencontainers.image.source https://github.com/flyteorg/flyte/

COPY --from=builder /artifacts /bin

# Ensure the latest CA certs are present to authenticate SSL connections.
RUN apk --update add ca-certificates

RUN addgroup -S flyte && adduser -S flyte -G flyte
USER flyte

CMD ["artifacts"]
1 change: 1 addition & 0 deletions Dockerfile.sandbox-lite
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ WORKDIR /app/flyte

COPY datacatalog datacatalog
COPY flyteadmin flyteadmin
COPY flyteartifacts flyteartifacts
COPY flytecopilot flytecopilot
COPY flyteidl flyteidl
COPY flyteplugins flyteplugins
Expand Down
19 changes: 19 additions & 0 deletions charts/flyte-sandbox/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,25 @@ A Helm chart for the Flyte local sandbox
| flyte-binary.clusterResourceTemplates.inlineConfigMap | string | `"{{ include \"flyte-sandbox.clusterResourceTemplates.inlineConfigMap\" . }}"` | |
| flyte-binary.configuration.database.host | string | `"{{ printf \"%s-postgresql\" .Release.Name | trunc 63 | trimSuffix \"-\" }}"` | |
| flyte-binary.configuration.database.password | string | `"postgres"` | |
| flyte-binary.configuration.inline.artifactsProcessor.cloudProvider | string | `"Sandbox"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.config.access_key_id | string | `"minio"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.config.auth_type | string | `"accesskey"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.config.disable_ssl | bool | `true` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.config.endpoint | string | `"http://flyte-sandbox-minio.flyte:9000"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.config.secret_key | string | `"miniostorage"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.config.v2_signing | bool | `true` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.stow.kind | string | `"s3"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactBlobStoreConfig.type | string | `"stow"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactDatabaseConfig.postgres.dbname | string | `"artifacts"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactDatabaseConfig.postgres.host | string | `"{{ printf \"%s-postgresql\" .Release.Name | trunc 63 | trimSuffix \"-\" }}"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactDatabaseConfig.postgres.options | string | `"sslmode=disable"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactDatabaseConfig.postgres.password | string | `"postgres"` | |
| flyte-binary.configuration.inline.artifactsServer.artifactDatabaseConfig.postgres.port | int | `5432` | |
| flyte-binary.configuration.inline.artifactsServer.artifactDatabaseConfig.postgres.username | string | `"postgres"` | |
| flyte-binary.configuration.inline.cloudEvents.cloudEventVersion | string | `"v2"` | |
| flyte-binary.configuration.inline.cloudEvents.enable | bool | `true` | |
| flyte-binary.configuration.inline.cloudEvents.type | string | `"sandbox"` | |
| flyte-binary.configuration.inline.flyteadmin.featureGates.enableArtifacts | bool | `true` | |
| flyte-binary.configuration.inline.plugins.k8s.default-env-vars[0].FLYTE_AWS_ENDPOINT | string | `"http://{{ printf \"%s-minio\" .Release.Name | trunc 63 | trimSuffix \"-\" }}.{{ .Release.Namespace }}:9000"` | |
| flyte-binary.configuration.inline.plugins.k8s.default-env-vars[1].FLYTE_AWS_ACCESS_KEY_ID | string | `"minio"` | |
| flyte-binary.configuration.inline.plugins.k8s.default-env-vars[2].FLYTE_AWS_SECRET_ACCESS_KEY | string | `"miniostorage"` | |
Expand Down
6 changes: 6 additions & 0 deletions charts/flyte-sandbox/templates/local/endpoint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,10 @@ subsets:
- name: webhook
port: 9443
protocol: TCP
- name: artifact-http
port: 50050
protocol: TCP
- name: artifact-grpc
port: 50051
protocol: TCP
{{- end }}
6 changes: 6 additions & 0 deletions charts/flyte-sandbox/templates/local/service.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,10 @@ spec:
- name: webhook
port: 9443
protocol: TCP
- name: artifact-http
port: 50050
protocol: TCP
- name: artifact-grpc
port: 50051
protocol: TCP
{{- end }}
37 changes: 36 additions & 1 deletion charts/flyte-sandbox/templates/proxy/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,10 @@ data:
prefix: "/v1"
route:
cluster: flyte
- match:
prefix: "/artifacts/api"
route:
cluster: artifact_http
- match:
prefix: "/flyteidl.service.AdminService"
route:
Expand All @@ -108,6 +112,10 @@ data:
prefix: "/flyteidl.service.SignalService"
route:
cluster: flyte_grpc
- match:
prefix: "/flyteidl.artifact.ArtifactRegistry"
route:
cluster: artifact_grpc
{{- end }}
{{- if index .Values "kubernetes-dashboard" "enabled" }}
- match:
Expand Down Expand Up @@ -172,6 +180,34 @@ data:
address: {{ include "flyte-sandbox.localHeadlessService" . }}
{{- end }}
port_value: 8089
- name: artifact_http
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: artifact_http
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: flyte-sandbox-local
port_value: 50050
- name: artifact_grpc
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: artifact_grpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: flyte-sandbox-local
port_value: 50051
{{- end }}
{{- if index .Values "kubernetes-dashboard" "enabled" }}
- name: kubernetes-dashboard
Expand Down Expand Up @@ -203,5 +239,4 @@ data:
address: {{ .Release.Name }}-minio
port_value: 9001
{{- end }}

{{- end }}
29 changes: 29 additions & 0 deletions charts/flyte-sandbox/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,35 @@ flyte-binary:
ephemeralStorage: 0
gpu: 0
memory: 0
cloudEvents:
enable: true
cloudEventVersion: v2
type: sandbox
flyteadmin:
featureGates:
enableArtifacts: true
artifactsServer:
artifactBlobStoreConfig:
type: stow
stow:
kind: s3
config:
disable_ssl: true
v2_signing: true
endpoint: http://flyte-sandbox-minio.flyte:9000
auth_type: accesskey
access_key_id: minio
secret_key: miniostorage
artifactDatabaseConfig:
postgres:
username: postgres
password: postgres
host: '{{ printf "%s-postgresql" .Release.Name | trunc 63 | trimSuffix "-" }}'
port: 5432
dbname: artifacts
options: "sslmode=disable"
artifactsProcessor:
cloudProvider: Sandbox
storage:
signedURL:
stowConfigOverride:
Expand Down
5 changes: 5 additions & 0 deletions cmd/single/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,18 @@ type Config struct {
Propeller Propeller `json:"propeller" pflag:",Configuration to disable propeller or any of its components."`
Admin Admin `json:"admin" pflag:",Configuration to disable FlyteAdmin or any of its components"`
DataCatalog DataCatalog `json:"dataCatalog" pflag:",Configuration to disable DataCatalog or any of its components"`
Artifact Artifacts `json:"artifact" pflag:",Configuration to disable Artifact or any of its components"`
}

type Propeller struct {
Disabled bool `json:"disabled" pflag:",Disables flytepropeller in the single binary mode"`
DisableWebhook bool `json:"disableWebhook" pflag:",Disables webhook only"`
}

type Artifacts struct {
Disabled bool `json:"disabled" pflag:",Disables flyteartifacts in the single binary mode"`
}

type Admin struct {
Disabled bool `json:"disabled" pflag:",Disables flyteadmin in the single binary mode"`
DisableScheduler bool `json:"disableScheduler" pflag:",Disables Native scheduler in the single binary mode"`
Expand Down
47 changes: 47 additions & 0 deletions cmd/single/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@ package single

import (
"context"
sharedCmd "github.com/flyteorg/flyte/flyteartifacts/cmd/shared"
"github.com/flyteorg/flyte/flyteartifacts/pkg/configuration"
artifactsServer "github.com/flyteorg/flyte/flyteartifacts/pkg/server"
"github.com/flyteorg/flyte/flytestdlib/database"
"net/http"
"os"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
Expand Down Expand Up @@ -63,6 +67,39 @@ func startClusterResourceController(ctx context.Context) error {
return nil
}

func startArtifact(ctx context.Context, cfg Artifacts) error {
if cfg.Disabled {
logger.Infof(ctx, "Artifacts server is disabled. Skipping...")
return nil
}
// Roughly copies main/NewMigrateCmd
logger.Infof(ctx, "Artifacts: running database migrations if any...")
migs := artifactsServer.GetMigrations(ctx)
dbConfig := artifactsServer.GetDbConfig()
err := database.Migrate(context.Background(), dbConfig, migs)
if err != nil {
logger.Errorf(ctx, "Failed to run Artifacts database migrations. Error: %v", err)
return err
}

g, childCtx := errgroup.WithContext(ctx)

// Rough copy of NewServeCmd
g.Go(func() error {
cfg := configuration.GetApplicationConfig()
serverCfg := &cfg.ArtifactServerConfig
err := sharedCmd.ServeGateway(childCtx, "artifacts", serverCfg, artifactsServer.GrpcRegistrationHook,
artifactsServer.HttpRegistrationHook)
if err != nil {
logger.Errorf(childCtx, "Failed to start Artifacts server. Error: %v", err)
return err
}
return nil
})

return g.Wait()
}

func startAdmin(ctx context.Context, cfg Admin) error {
logger.Infof(ctx, "Running Database Migrations...")
if err := adminServer.Migrate(ctx); err != nil {
Expand Down Expand Up @@ -232,6 +269,16 @@ var startCmd = &cobra.Command{
})
}

if !cfg.Artifact.Disabled {
g.Go(func() error {
err := startArtifact(childCtx, cfg.Artifact)
if err != nil {
logger.Panicf(childCtx, "Failed to start Artifacts server, err: %v", err)
}
return nil
})
}

if !cfg.Propeller.Disabled {
g.Go(func() error {
err := startPropeller(childCtx, cfg.Propeller)
Expand Down
2 changes: 1 addition & 1 deletion datacatalog/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ require (
k8s.io/klog/v2 v2.100.1 // indirect
k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9 // indirect
k8s.io/utils v0.0.0-20230406110748-d93618cff8a2 // indirect
sigs.k8s.io/controller-runtime v0.0.0-00010101000000-000000000000 // indirect
sigs.k8s.io/controller-runtime v0.16.2 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
Expand Down
15 changes: 13 additions & 2 deletions docker/sandbox-bundled/Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
define FLYTE_BINARY_BUILD
mkdir -p images/tar/$(1)

docker buildx build \
docker buildx build --ssh default \
--build-arg FLYTECONSOLE_VERSION=$(FLYTECONSOLE_VERSION) \
--platform linux/$(1) \
--tag flyte-binary:sandbox \
Expand Down Expand Up @@ -41,9 +41,20 @@ build: flyte manifests
--driver docker-container --driver-opt image=moby/buildkit:master \
--buildkitd-flags '--allow-insecure-entitlement security.insecure' \
--platform linux/arm64,linux/amd64
docker buildx build --builder flyte-sandbox --allow security.insecure --load \
docker buildx build --ssh default --builder flyte-sandbox --allow security.insecure --load \
--tag flyte-sandbox:latest .

# This is here because we want to be able to push locally, not depend on GH actions
.PHONY: build_push
build_push: flyte manifests
[ -n "$(shell docker buildx ls | awk '/^flyte-sandbox / {print $$1}')" ] || \
docker buildx create --name flyte-sandbox \
--driver docker-container --driver-opt image=moby/buildkit:master \
--buildkitd-flags '--allow-insecure-entitlement security.insecure' \
--platform linux/arm64,linux/amd64
docker buildx build --ssh default --builder flyte-sandbox --allow security.insecure --push \
--tag ghcr.io/flyteorg/flyte-sandbox:a_v0.1.1 .

# Port map
# 6443 - k8s API server
# 30000 - Docker Registry
Expand Down
Loading