Skip to content

Commit

Permalink
Single binary (flyteorg#2265)
Browse files Browse the repository at this point in the history
* wip

Signed-off-by: Ketan Umare <[email protected]>

* WIP, almost there

Signed-off-by: Ketan Umare <[email protected]>

* wip

Signed-off-by: Ketan Umare <[email protected]>

* working

Signed-off-by: Ketan Umare <[email protected]>

* Using go:embed to embed console into the go binary

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* Not working (added datacatalog)

Signed-off-by: Ketan Umare <[email protected]>

* Updated console.go

Signed-off-by: Kevin Su <[email protected]>

* working

Signed-off-by: Ketan Umare <[email protected]>

* updated

Signed-off-by: Ketan Umare <[email protected]>

* working /console/

Signed-off-by: Ketan Umare <[email protected]>

* Updated console - this time working

Signed-off-by: Ketan Umare <[email protected]>

* Add dockerfile to build sandbox-lite

Signed-off-by: Kevin Su <[email protected]>

* Updated dockerfile

Signed-off-by: Kevin Su <[email protected]>

* added unit test

Signed-off-by: Ketan Umare <[email protected]>

* Updated dockerfile

Signed-off-by: Kevin Su <[email protected]>

* Updated helm chart

Signed-off-by: Kevin Su <[email protected]>

* Updated grpc port

Signed-off-by: Kevin Su <[email protected]>

* Updated helm docs

Signed-off-by: Kevin Su <[email protected]>

* updated config

Signed-off-by: Kevin Su <[email protected]>

* updated datacatalog

Signed-off-by: Kevin Su <[email protected]>

* updated admin

Signed-off-by: Kevin Su <[email protected]>

* Fixed tests

Signed-off-by: Kevin Su <[email protected]>

* Updated dependencies

Signed-off-by: Kevin Su <[email protected]>

* Push sandbox-lite in github action

Signed-off-by: Kevin Su <[email protected]>

* Ability to disable certain things in single binary mode

Signed-off-by: Ketan Umare <[email protected]>

* Updated dockerfile

Signed-off-by: Kevin Su <[email protected]>

* helm chart and updated docker file

Signed-off-by: Kevin Su <[email protected]>

* Webhook

Signed-off-by: Kevin Su <[email protected]>

* webhook config

Signed-off-by: Kevin Su <[email protected]>

* remove metricsBindAddress

Signed-off-by: Kevin Su <[email protected]>

* fix helm tests

Signed-off-by: Kevin Su <[email protected]>

* Updated propeller

Signed-off-by: Kevin Su <[email protected]>

* Configure serviceHttpEndpoint

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* Update flyte components versions

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* wip

Signed-off-by: Haytham Abuelfutuh <[email protected]>

* update datacatalog version

Signed-off-by: Katrina Rogan <[email protected]>

* set go version for generating flyte manifest

Signed-off-by: Katrina Rogan <[email protected]>

* debug

Signed-off-by: Katrina Rogan <[email protected]>

* Fix error

Signed-off-by: Kevin Su <[email protected]>

* Update propeller

Signed-off-by: Kevin Su <[email protected]>

* Update propeller

Signed-off-by: Kevin Su <[email protected]>

* revert

Signed-off-by: Katrina Rogan <[email protected]>

* update GA

Signed-off-by: Kevin Su <[email protected]>

* Removed storage chart

Signed-off-by: Yuvraj <[email protected]>

* Refactor dockerfile

Signed-off-by: Yuvraj <[email protected]>

* Added postgres port in docker file

Signed-off-by: Yuvraj <[email protected]>

* more changes

Signed-off-by: Yuvraj <[email protected]>

* fix docker restart

Signed-off-by: Yuvraj <[email protected]>

* added namespace template in helm deps chart

Signed-off-by: Yuvraj <[email protected]>

* Disable contour and make helm

Signed-off-by: Kevin Su <[email protected]>

* Disable contour and make helm

Signed-off-by: Kevin Su <[email protected]>

* Updated flyteconsole

Signed-off-by: Kevin Su <[email protected]>

* Updated flyteadmin

Signed-off-by: Kevin Su <[email protected]>

* typo

Signed-off-by: Kevin Su <[email protected]>

* Added template and make helm

Signed-off-by: Kevin Su <[email protected]>

* Updated template

Signed-off-by: Kevin Su <[email protected]>

* Updated alpine

Signed-off-by: Kevin Su <[email protected]>

Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Haytham Abuelfutuh <[email protected]>
Co-authored-by: Katrina Rogan <[email protected]>
Co-authored-by: Yuvraj <[email protected]>
  • Loading branch information
5 people authored Apr 12, 2022
1 parent 2a46122 commit 0bb8568
Show file tree
Hide file tree
Showing 26 changed files with 3,526 additions and 3,229 deletions.
63 changes: 63 additions & 0 deletions .github/workflows/sandbox.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,3 +74,66 @@ jobs:
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
sandbox-lite-build-dind:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: "0"
- name: Set flyte version to release
id: set_version
run: |
if [ ${{ github.event_name}} = "release" ]; then
echo ::set-output name=flyte_version::$(echo ${{ github.event.release.tag_name }})
else
echo ::set-output name=flyte_version::latest
fi
- name: Prepare DIND Image Names
id: dind-names
uses: docker/metadata-action@v3
with:
# list of Docker images to use as base name for tags
images: |
ghcr.io/${{ github.repository_owner }}/flyte-sandbox-lite
tags: |
${{ steps.set_version.outputs.flyte_version }}
type=sha,format=long
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
- name: Cache Docker layers
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-single-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-single-buildx
- name: Login to GitHub Container Registry
if: ${{ github.event_name == 'release' }}
uses: docker/login-action@v1
with:
registry: ghcr.io
username: "${{ secrets.FLYTE_BOT_USERNAME }}"
password: "${{ secrets.FLYTE_BOT_PAT }}"
- name: Build and push DIND Image
uses: docker/build-push-action@v2
with:
context: .
platforms: linux/arm64, linux/amd64
push: ${{ github.event_name == 'release' }}
target: dind
tags: ${{ steps.dind-names.outputs.tags }}
build-args: "FLYTE_VERSION=${{ steps.set_version.outputs.flyte_version }}"
file: docker/sandbox-lite/Dockerfile
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,mode=max,dest=/tmp/.buildx-cache-new
- # Temp fix
# https://github.com/docker/build-push-action/issues/252
# https://github.com/moby/buildkit/issues/1896
name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@ __pycache__/
/charts/flyte/Chart.lock
/charts/flyte-deps/Chart.lock
/charts/flyte-core/Chart.lock
/docker/sandbox-lite/storage/charts
/docker/sandbox-lite/storage/Chart.lock
**/*.bak-
bin/-
bin/
.python-version
dist/
dist
*.db
flyte
vendor/
14 changes: 14 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,20 @@ define PIP_COMPILE
pip-compile $(1) --upgrade --verbose
endef

GIT_VERSION := $(shell git describe --always --tags)
GIT_HASH := $(shell git rev-parse --short HEAD)
TIMESTAMP := $(shell date '+%Y-%m-%d')
PACKAGE ?=github.com/flyteorg/flytestdlib
LD_FLAGS="-s -w -X $(PACKAGE)/version.Version=$(GIT_VERSION) -X $(PACKAGE)/version.Build=$(GIT_HASH) -X $(PACKAGE)/version.BuildTime=$(TIMESTAMP)"

.PHONY: compile
compile:
go build -o flyte -ldflags=$(LD_FLAGS) ./cmd/ && mv ./flyte ${GOPATH}/bin

.PHONY: linux_compile
linux_compile:
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o /artifacts/flyte -ldflags=$(LD_FLAGS) ./cmd/

.PHONY: update_boilerplate
update_boilerplate:
@boilerplate/update.sh
Expand Down
6 changes: 3 additions & 3 deletions charts/flyte-deps/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ helm upgrade -f values.yaml flyte .
| contour.contour.resources | object | `{"limits":{"cpu":"100m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}` | Default resources requests and limits for Contour |
| contour.contour.resources.limits | object | `{"cpu":"100m","memory":"100Mi"}` | Limits are the maximum set of resources needed for this pod |
| contour.contour.resources.requests | object | `{"cpu":"10m","memory":"50Mi"}` | Requests are the minimum set of resources needed for this pod |
| contour.enabled | bool | `true` | - enable or disable Contour deployment installation |
| contour.enabled | bool | `false` | - enable or disable Contour deployment installation |
| contour.envoy.resources | object | `{"limits":{"cpu":"100m","memory":"100Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}` | Default resources requests and limits for Envoy |
| contour.envoy.resources.limits | object | `{"cpu":"100m","memory":"100Mi"}` | Limits are the maximum set of resources needed for this pod |
| contour.envoy.resources.requests | object | `{"cpu":"10m","memory":"50Mi"}` | Requests are the minimum set of resources needed for this pod |
Expand Down Expand Up @@ -97,7 +97,7 @@ helm upgrade -f values.yaml flyte .
| postgres.podAnnotations | object | `{}` | Annotations for Postgres pods |
| postgres.replicaCount | int | `1` | Replicas count for Postgres deployment |
| postgres.resources | object | `{"limits":{"cpu":"1000m","memory":"512Mi"},"requests":{"cpu":"10m","memory":"128Mi"}}` | Default resources requests and limits for Postgres deployment |
| postgres.service | object | `{"annotations":{},"type":"ClusterIP"}` | Service settings for Postgres |
| postgres.service | object | `{"annotations":{},"type":"NodePort"}` | Service settings for Postgres |
| postgres.tolerations | list | `[]` | tolerations for Postgres deployment |
| redis.enabled | bool | `false` | - enable or disable Redis Statefulset installation |
| redoc.affinity | object | `{}` | affinity for Minio deployment |
Expand All @@ -115,5 +115,5 @@ helm upgrade -f values.yaml flyte .
| redoc.tolerations | list | `[]` | tolerations for Minio deployment |
| sparkoperator | object | `{"enabled":false}` | Optional: Spark Plugin using the Spark Operator |
| sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation |
| webhook.enabled | bool | `false` | |
| webhook.enabled | bool | `true` | |
| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook |
2 changes: 2 additions & 0 deletions charts/flyte-deps/templates/redoc/deployment.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{{- if .Values.redoc.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
Expand Down Expand Up @@ -42,4 +43,5 @@ spec:
{{- with .Values.redoc.tolerations }}
tolerations: {{ tpl (toYaml .) $ | nindent 8 }}
{{- end }}
{{- end }}

6 changes: 3 additions & 3 deletions charts/flyte-deps/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ redis:
enabled: false

webhook:
enabled: false
enabled: true
# -- Service settings for the webhook
service:
annotations:
Expand Down Expand Up @@ -37,7 +37,7 @@ postgres:
# -- Service settings for Postgres
service:
annotations: {}
type: ClusterIP
type: NodePort
# -- Annotations for Postgres pods
podAnnotations: {}
# -- nodeSelector for Postgres deployment
Expand Down Expand Up @@ -126,7 +126,7 @@ redoc:

contour:
# --- enable or disable Contour deployment installation
enabled: true
enabled: false
# -- Replicas count for Contour deployment
replicaCount: 1
contour:
Expand Down
2 changes: 1 addition & 1 deletion charts/flyte/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,7 +223,7 @@ helm upgrade -f values-sandbox.yaml flyte .
| postgres.podAnnotations | object | `{}` | Annotations for Postgres pods |
| postgres.replicaCount | int | `1` | Replicas count for Postgres deployment |
| postgres.resources | object | `{"limits":{"cpu":"1000m","memory":"512Mi"},"requests":{"cpu":"10m","memory":"128Mi"}}` | Default resources requests and limits for Postgres deployment |
| postgres.service | object | `{"annotations":{},"type":"ClusterIP"}` | Service settings for Postgres |
| postgres.service | object | `{"annotations":{},"type":"NodePort"}` | Service settings for Postgres |
| postgres.tolerations | list | `[]` | tolerations for Postgres deployment |
| redis | object | `{"enabled":false}` | --------------------------------------------- REDIS SETTINGS |
| redis.enabled | bool | `false` | - enable or disable Redis Statefulset installation |
Expand Down
1 change: 1 addition & 0 deletions charts/flyte/templates/postgres/service.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ spec:
{{- end }}
ports:
- name: postgres
nodePort: 30089
port: 5432
protocol: TCP
targetPort: postgres
Expand Down
2 changes: 1 addition & 1 deletion charts/flyte/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -693,7 +693,7 @@ postgres:
# -- Service settings for Postgres
service:
annotations: {}
type: ClusterIP
type: NodePort
# -- Annotations for Postgres pods
podAnnotations: {}
# -- nodeSelector for Postgres deployment
Expand Down
15 changes: 15 additions & 0 deletions cmd/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package main

import (
"github.com/flyteorg/flyte/cmd/single"
_ "github.com/flyteorg/flytepropeller/plugins"
"github.com/golang/glog"
)

func main() {
glog.V(2).Info("Starting Flyte")
err := single.Execute()
if err != nil {
panic(err)
}
}
36 changes: 36 additions & 0 deletions cmd/single/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
package single

import "github.com/flyteorg/flytestdlib/config"

//go:generate pflags Config --default-var=DefaultConfig

var (
DefaultConfig = &Config{}
configSection = config.MustRegisterSection("flyte", DefaultConfig)
)

type Config struct {
Propeller Propeller `json:"propeller" pflag:",Configuration to disable propeller or any of its components."`
Admin Admin `json:"admin" pflag:",Configuration to disable FlyteAdmin or any of its components"`
DataCatalog DataCatalog `json:"data_catalog" pflag:",Configuration to disable DataCatalog or any of its components"`
}

type Propeller struct {
Disabled bool `json:"disabled" pflag:",Disables flytepropeller in the single binary mode"`
DisableWebhook bool `json:"disable_webhook" pflag:",Disables webhook only"`
}

type Admin struct {
Disabled bool `json:"disabled" pflag:",Disables flyteadmin in the single binary mode"`
DisableScheduler bool `json:"disable_scheduler" pflag:",Disables Native scheduler in the single binary mode"`
DisableClusterResourceManager bool `json:"disable_cluster_resource_manager" pflag:",Disables Cluster resource manager"`
}

type DataCatalog struct {
Disabled bool `json:"disabled" pflag:",Disables datacatalog in the single binary mode"`
}

// GetConfig returns a handle to the configuration for Flyte Single Binary
func GetConfig() *Config {
return configSection.GetConfig().(*Config)
}
66 changes: 66 additions & 0 deletions cmd/single/console.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package single

import (
"embed"
"net/http"
"strings"
)

//go:embed dist/*
var console embed.FS

const consoleRoot = "/console"
const assetsDir = "assets"
const packageDir = "dist"
const indexHTML = "/index.html"

// GetConsoleFile returns the console file that should be used for the given path.
// Every path has a '/console' as the prefix. After dropping this prefix, the following 3 rules are checked
// Rule 1: If path is now "" or "/" then return index.html
// Rule 2: If path contains no "assets" sub-string and has an additional substring "/", then return "index.html".
// This is to allow vanity urls in React
// Rule 3: Finally return every file path as is
// For every file add a "dist" as the prefix, as every file is assumed to be packaged in "dist" folder fv
func GetConsoleFile(name string) string {
name = strings.TrimPrefix(name, consoleRoot)
name = strings.TrimPrefix(name, "/"+assetsDir)
if name == "" || name == "/" {
name = indexHTML
} else if !strings.Contains(name, assetsDir) {
if strings.Contains(name[1:], "/") {
name = indexHTML
}
}
return packageDir + name
}

type consoleFS struct {
fs http.FileSystem
}

// Open Implements a specific handler for Console - SinglePage React App
// the path re-writing is critical
func (f consoleFS) Open(name string) (http.File, error) {
return f.fs.Open(GetConsoleFile(name))
}

// GetConsoleHandlers returns a set of handlers that can be added to the Server mux and can handle all console related
// requests
func GetConsoleHandlers() map[string]func(http.ResponseWriter, *http.Request) {
handlers := make(map[string]func(http.ResponseWriter, *http.Request))
// Serves console
consoleHandler := http.FileServer(consoleFS{fs: http.FS(console)})

// This is the base handle for "/console"
handlers[consoleRoot] = func(writer http.ResponseWriter, request *http.Request) {
consoleHandler.ServeHTTP(writer, request)
}

// this is the root handler for any pattern that matches "/console/
// http.mux needs a trailing "/" to allow longest pattern matching.
// For the previous handler "/console" the mux will only consider exact match
handlers[consoleRoot+"/"] = func(writer http.ResponseWriter, request *http.Request) {
consoleHandler.ServeHTTP(writer, request)
}
return handlers
}
24 changes: 24 additions & 0 deletions cmd/single/console_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
package single

import "testing"

func TestGetConsoleFile(t *testing.T) {
tests := []struct {
name string
want string
}{
{"/console", "dist/index.html"},
{"/console/", "dist/index.html"},
{"/console/main.js", "dist/main.js"},
{"/console/assets/xyz.png", "dist/assets/xyz.png"},
{"/console/assets/dir/xyz.png", "dist/assets/dir/xyz.png"},
{"console/projects/flytesnacks/workflows?domain=development", "dist/index.html"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := GetConsoleFile(tt.name); got != tt.want {
t.Errorf("GetConsoleFile() = %v, want %v", got, tt.want)
}
})
}
}
Loading

0 comments on commit 0bb8568

Please sign in to comment.