Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Agent Metadata Servicer #4511

Merged
merged 63 commits into from
Dec 20, 2023
Merged
Show file tree
Hide file tree
Changes from 61 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
db7a34e
proto
Dec 1, 2023
f6a9e3b
update pingsu's pr
Dec 2, 2023
8f2869c
new generate
Dec 2, 2023
ccaf9f2
Merge branch 'master' of https://github.com/Future-Outlier/flyte into…
Dec 2, 2023
4ce0955
add supported task type
Dec 4, 2023
9eb72d6
change os FileMode from 511 to 420
Dec 4, 2023
637e13e
add test
Dec 4, 2023
02400ef
add initialTaskTypes
Dec 4, 2023
1e8b98e
list all agents
Dec 4, 2023
7d77df5
update config supported task type
Dec 4, 2023
77f0903
combineToDistinctTaskTypes
Dec 4, 2023
cbc8eb5
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 4, 2023
b14da0f
support all agents supported task type
Dec 4, 2023
5db9e38
add all agents supported task type
Dec 4, 2023
8756f20
sync plugin routing mechansim
Dec 4, 2023
9eecd67
need refactor and add test
Dec 4, 2023
ada503d
Merge branch 'master' of https://github.com/Future-Outlier/flyte into…
Dec 5, 2023
dcff3c3
solve merge conflict
Dec 5, 2023
7996769
merge conflict
Dec 5, 2023
8f28ffd
Update flyteplugins/go/tasks/plugins/webapi/agent/plugin.go
Future-Outlier Dec 6, 2023
d764139
Merge branch 'master' of https://github.com/Future-Outlier/flyte into…
Dec 10, 2023
ef02107
refactor
Dec 10, 2023
aa24087
remove tags
Dec 10, 2023
9f14c08
add small test
Dec 10, 2023
100e985
Merge branch 'master' of https://github.com/Future-Outlier/flyte into…
Dec 12, 2023
c657e0e
Add agent registry
pingsutw Dec 12, 2023
ce7647f
Merge branch 'agent-metadata-proto-service' of https://github.com/Fut…
Dec 12, 2023
806a602
need to change supported task type to supported task types
Dec 12, 2023
45c1b4d
refactor agentDeployments
Dec 12, 2023
09daecd
change back proto
Dec 12, 2023
fb94d38
fileMode
Dec 12, 2023
b0daff8
Better annotation
Dec 12, 2023
37095e9
fix test
Dec 12, 2023
1228f15
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 12, 2023
c8d78eb
add agent metadata test
Dec 12, 2023
8167f27
Merge branch 'agent-metadata-proto-service' of https://github.com/Fut…
Dec 12, 2023
6930543
fix load plugin bug
Dec 12, 2023
2bf45bd
change default agent
Dec 13, 2023
29a2bd4
more readable
Dec 13, 2023
c429e13
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 13, 2023
2c285c6
move back get client func
Dec 13, 2023
9755ab4
remove secret
Dec 16, 2023
7cd33d2
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 16, 2023
425b54a
remove secrets in agent.proto
Dec 17, 2023
cb7ea4c
Merge branch 'agent-metadata-proto-service' of https://github.com/Fut…
Dec 17, 2023
b3f7c76
extend sandbox-bundled-functional-tests timeout minutes
Dec 17, 2023
de33fc3
Update flyteplugins/go/tasks/plugins/webapi/agent/plugin.go
Future-Outlier Dec 17, 2023
9e5f835
fix bug
Dec 17, 2023
9c8e3cf
test disable-agent
Dec 17, 2023
44d1d53
revert --disable-agent
Dec 17, 2023
1cd0e48
test
Dec 18, 2023
ef9cfe8
use grpc code status to handle case list agent method not implemented
Dec 19, 2023
15ec587
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 19, 2023
e76d81f
rename grpc_status to grpcStatus
Dec 19, 2023
2bb181a
Merge branch 'agent-metadata-proto-service' of https://github.com/Fut…
Dec 19, 2023
78eeefd
remove is_sync proto
Dec 19, 2023
1d512e6
remove is_sync
Dec 19, 2023
6432585
Update complete-agent.yaml
pingsutw Dec 19, 2023
9b33ccf
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 19, 2023
3a4ae5c
Merge branch 'master' into agent-metadata-proto-service
Future-Outlier Dec 19, 2023
3b07407
change getFinalAgent implementation and interface
Dec 19, 2023
8f46477
Update flyteidl/protos/flyteidl/service/agent.proto
Future-Outlier Dec 20, 2023
c1e3492
rename ListAgent to ListAgents method
Dec 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/single-binary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ jobs:
env:
FLYTESNACKS_PRIORITIES: "P0"
FLYTESNACKS_VERSION: ""
timeout-minutes: 20
timeout-minutes: 60
needs: [build-and-push-single-binary-image]
steps:
- name: Set latest Flytesnacks release
Expand Down
9 changes: 1 addition & 8 deletions charts/flyte-binary/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -61,16 +61,9 @@ data:
k8s-array:
logs:
config: {{- include "flyte-binary.configuration.logging.plugins" . | nindent 12 }}
{{- if .Values.configuration.agentService }}
{{- if .Values.flyteagent.enabled }}
agent-service:
{{- tpl ( .Values.configuration.agentService | toYaml ) . | nindent 8 }}
{{- else if .Values.flyteagent.enabled }}
agent-service:
defaultAgent:
endpoint: {{ include "flyteagent.name" . }}:{{ include "flyteagent.servicePort" . }}
insecure: true
supportedTaskTypes:
- bigquery_query_job_task
{{- end }}
002-database.yaml: |
{{- with .Values.configuration.database }}
Expand Down
1 change: 0 additions & 1 deletion charts/flyte-binary/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,6 @@ configuration:
timeouts:
GetTask: 10s
defaultTimeout: 10s
# Uncomment and modify to include configuration for Flyte Agent
supportedTaskTypes:
- default_task
# externalConfigMap Specify an existing, external ConfigMap to use as configuration for Flyte
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -123,15 +123,6 @@ data:
kubernetes-enabled: false
cloudwatch-enabled: false
stackdriver-enabled: false
agent-service:
defaultAgent:
defaultTimeout: 10s
endpoint: dns:///flyteagent.flyte.svc.cluster.local:8000
insecure: true
timeouts:
GetTask: 10s
supportedTaskTypes:
- default_task
002-database.yaml: |
database:
postgres:
Expand Down Expand Up @@ -367,7 +358,7 @@ spec:
app.kubernetes.io/instance: flyte
app.kubernetes.io/component: flyte-binary
annotations:
checksum/configuration: 51494d8c2f3a6df2711d789217155f1e1b60e782dcbfd4564bfcccda24700576
checksum/configuration: 94a584c983cfe1077f02569ae00f81c4fddae4505901006eda76d9338bd3dda2
checksum/configuration-secret: d5d93f4e67780b21593dc3799f0f6682aab0765e708e4020939975d14d44f929
checksum/cluster-resource-templates: 7dfa59f3d447e9c099b8f8ffad3af466fecbc9cf9f8c97295d9634254a55d4ae
spec:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@ helmCharts:
- name: flyte-sandbox
releaseName: flyte-sandbox
namespace: flyte
- name: flyteagent
releaseName: flyteagent
namespace: flyte
valuesInline:
flyte-binary:
flyteagent:
enabled: True
namespace: flyte
resources:
- ../namespace.yaml

patchesStrategicMerge:
- patch.yaml
- add-agent-secret-patch.yaml
16 changes: 8 additions & 8 deletions docker/sandbox-bundled/manifests/complete-agent.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: flyteagent
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: flyteagent
helm.sh/chart: flyteagent-v0.1.10
Expand Down Expand Up @@ -816,7 +816,7 @@ type: Opaque
---
apiVersion: v1
data:
haSharedSecret: ZnNFaHlpM01FZjRaQmNJSA==
haSharedSecret: cHNOS0lPZmw0andDczdaaA==
proxyPassword: ""
proxyUsername: ""
kind: Secret
Expand Down Expand Up @@ -1125,7 +1125,7 @@ metadata:
annotations:
projectcontour.io/upstream-protocol.h2c: grpc
labels:
app.kubernetes.io/instance: flyteagent
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: flyteagent
helm.sh/chart: flyteagent-v0.1.10
Expand All @@ -1138,7 +1138,7 @@ spec:
protocol: TCP
targetPort: agent-grpc
selector:
app.kubernetes.io/instance: flyteagent
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/name: flyteagent
type: ClusterIP
---
Expand Down Expand Up @@ -1411,7 +1411,7 @@ spec:
metadata:
annotations:
checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81
checksum/secret: 727daf2e5024ea4e541b87b717e6f9a8706552e0ae461f82b2e5260ad564766c
checksum/secret: ce47bbc1f3ca2d8673128f8efd226bd9c7ac4f32a6ded68e54e2ebb3a843065b
labels:
app: docker-registry
release: flyte-sandbox
Expand Down Expand Up @@ -1722,7 +1722,7 @@ apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/instance: flyteagent
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: flyteagent
helm.sh/chart: flyteagent-v0.1.10
Expand All @@ -1732,12 +1732,12 @@ spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: flyteagent
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/name: flyteagent
template:
metadata:
labels:
app.kubernetes.io/instance: flyteagent
app.kubernetes.io/instance: flyte-sandbox
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: flyteagent
helm.sh/chart: flyteagent-v0.1.10
Expand Down
15 changes: 3 additions & 12 deletions docker/sandbox-bundled/manifests/complete.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -465,15 +465,6 @@ data:
kubernetes-template-uri: http://localhost:30080/kubernetes-dashboard/#/log/{{.namespace }}/{{ .podName }}/pod?namespace={{ .namespace }}
cloudwatch-enabled: false
stackdriver-enabled: false
agent-service:
defaultAgent:
defaultTimeout: 10s
endpoint: dns:///flyteagent.flyte.svc.cluster.local:8000
insecure: true
timeouts:
GetTask: 10s
supportedTaskTypes:
- default_task
002-database.yaml: |
database:
postgres:
Expand Down Expand Up @@ -805,7 +796,7 @@ type: Opaque
---
apiVersion: v1
data:
haSharedSecret: UlI4aW9PaVFzcFpyRzhkTA==
haSharedSecret: RjR3Qnc0dWtGbHE3S3pnbA==
proxyPassword: ""
proxyUsername: ""
kind: Secret
Expand Down Expand Up @@ -1203,7 +1194,7 @@ spec:
metadata:
annotations:
checksum/cluster-resource-templates: 6fd9b172465e3089fcc59f738b92b8dc4d8939360c19de8ee65f68b0e7422035
checksum/configuration: 6755bdc74789d7e2a80161a9f811e41ea3c0cf12bab5cb0820c7c8539b4b88fe
checksum/configuration: e2233c2adb914be363ec4fae808ed60ffb230367503f73643f011e728b853b49
checksum/configuration-secret: 09216ffaa3d29e14f88b1f30af580d02a2a5e014de4d750b7f275cc07ed4e914
labels:
app.kubernetes.io/component: flyte-binary
Expand Down Expand Up @@ -1368,7 +1359,7 @@ spec:
metadata:
annotations:
checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81
checksum/secret: cb0b9e32c83fba67eefca5b8cd0a3375b07f5d5d29f6dab9a092f91ccf7bd200
checksum/secret: d027248867ca693de77339af17e89c22e7ad604b7abe616e40ca947ad0626a83
labels:
app: docker-registry
release: flyte-sandbox
Expand Down
4 changes: 2 additions & 2 deletions docker/sandbox-bundled/manifests/dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -499,7 +499,7 @@ metadata:
---
apiVersion: v1
data:
haSharedSecret: QzZJWkZlOTBQOUZxTUlvcA==
haSharedSecret: RWxBOE5BV0lIeEJ2d1c3ZA==
proxyPassword: ""
proxyUsername: ""
kind: Secret
Expand Down Expand Up @@ -933,7 +933,7 @@ spec:
metadata:
annotations:
checksum/config: 8f50e768255a87f078ba8b9879a0c174c3e045ffb46ac8723d2eedbe293c8d81
checksum/secret: dd543013ea0e17fb3c9b7be54fe7c6063e8f15ad6e0887596011ddd7f22f1540
checksum/secret: a2181c7387731100215f7ae94400f7330557b063f01f807d5aa2f333f266126e
labels:
app: docker-registry
release: flyte-sandbox
Expand Down
114 changes: 114 additions & 0 deletions flyteidl/clients/go/admin/mocks/AgentMetadataServiceClient.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading