forked from flyteorg/flyte
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Docs] Testing agents in the development environment (flyteorg#5106)
* Testing agents in the development environment Signed-off-by: Future-Outlier <[email protected]> * nit Signed-off-by: Future-Outlier <[email protected]> * nit Signed-off-by: Future-Outlier <[email protected]> * update Signed-off-by: Future-Outlier <[email protected]> * rename Signed-off-by: Future-Outlier <[email protected]> * blank Signed-off-by: Future-Outlier <[email protected]> * rerun build docs ci Signed-off-by: Future-Outlier <[email protected]> * update pingsu's advice Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Update pingsu's advice Signed-off-by: Future-Outlier <[email protected]> Co-authored-by: Kevin Su <[email protected]> * deploying agents in the sandbox Signed-off-by: Future-Outlier <[email protected]> * rename Signed-off-by: Future-Outlier <[email protected]> * nit Signed-off-by: Future-Outlier <[email protected]> * Implementing Agent Metadata Service Signed-off-by: Future-Outlier <[email protected]> * reorganize and copyedit new content Signed-off-by: nikki everett <[email protected]> --------- Signed-off-by: Future-Outlier <[email protected]> Signed-off-by: nikki everett <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: nikki everett <[email protected]>
- Loading branch information
Showing
7 changed files
with
267 additions
and
22 deletions.
There are no files selected for viewing
98 changes: 98 additions & 0 deletions
98
docs/flyte_agents/deploying_agents_to_the_flyte_sandbox.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
--- | ||
jupytext: | ||
formats: md:myst | ||
text_representation: | ||
extension: .md | ||
format_name: myst | ||
--- | ||
|
||
(deploying_agents_to_the_flyte_sandbox)= | ||
# Deploying agents to the Flyte sandbox | ||
|
||
After you have finished {ref}`testing an agent locally <testing_agents_locally>`, you can deploy your agent to the Flyte sandbox. | ||
|
||
Here's a step by step guide to deploying your agent image to the Flyte sandbox. | ||
|
||
1. Start the Flyte sandbox: | ||
```bash | ||
flytectl demo start | ||
``` | ||
|
||
2. Build an agent image: | ||
You can go to [here](https://github.com/flyteorg/flytekit/blob/master/Dockerfile.agent) to see the Dockerfile we use in flytekit python. | ||
Take Databricks agent as an example: | ||
```Dockerfile | ||
FROM python:3.9-slim-bookworm | ||
|
||
RUN apt-get update && apt-get install build-essential git -y | ||
RUN pip install prometheus-client grpcio-health-checking | ||
RUN pip install --no-cache-dir -U flytekit \ | ||
git+https://github.com/flyteorg/flytekit.git@<gitsha>#subdirectory=plugins/flytekit-spark \ | ||
&& apt-get clean autoclean \ | ||
&& apt-get autoremove --yes \ | ||
&& rm -rf /var/lib/{apt,dpkg,cache,log}/ \ | ||
&& : | ||
|
||
CMD pyflyte serve agent --port 8000 | ||
``` | ||
```bash | ||
docker buildx build -t localhost:30000/flyteagent:example -f Dockerfile.agent . --load | ||
docker push localhost:30000/flyteagent:example | ||
``` | ||
|
||
2. Deploy your agent image to the Kubernetes cluster: | ||
```bash | ||
kubectl edit deployment flyteagent -n flyte | ||
``` | ||
Search for the `image` key and change its value to your agent image: | ||
```yaml | ||
image: localhost:30000/flyteagent:example | ||
``` | ||
3. Set up your secrets: | ||
Let's take Databricks agent as an example: | ||
```bash | ||
kubectl edit secret flyteagent -n flyte | ||
``` | ||
Get your `BASE64_ENCODED_DATABRICKS_TOKEN`: | ||
```bash | ||
echo -n "<DATABRICKS_TOKEN>" | base64 | ||
``` | ||
Add your token to the `data` field: | ||
```yaml | ||
apiVersion: v1 | ||
data: | ||
flyte_databricks_access_token: <BASE64_ENCODED_DATABRICKS_TOKEN> | ||
kind: Secret | ||
metadata: | ||
annotations: | ||
meta.helm.sh/release-name: flyteagent | ||
meta.helm.sh/release-namespace: flyte | ||
creationTimestamp: "2023-10-04T04:09:03Z" | ||
labels: | ||
app.kubernetes.io/managed-by: Helm | ||
name: flyteagent | ||
namespace: flyte | ||
resourceVersion: "753" | ||
uid: 5ac1e1b6-2a4c-4e26-9001-d4ba72c39e54 | ||
type: Opaque | ||
``` | ||
:::{note} | ||
Please ensure two things: | ||
1. The secret name consists only of lowercase English letters. | ||
2. The secret value is encoded in Base64. | ||
::: | ||
4. Restart development: | ||
```bash | ||
kubectl rollout restart deployment flyte-sandbox -n flyte | ||
``` | ||
|
||
5. Test your agent remotely in the Flyte sandbox: | ||
```bash | ||
pyflyte run --remote agent_workflow.py agent_task | ||
``` | ||
|
||
:::{note} | ||
You must build an image that includes the plugin for the task and specify its config with the [`--image` flag](https://docs.flyte.org/en/latest/api/flytekit/pyflyte.html#cmdoption-pyflyte-run-i) when running `pyflyte run` or in an {ref}`ImageSpec <imagespec>` definition in your workflow file. | ||
::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
48 changes: 48 additions & 0 deletions
48
docs/flyte_agents/implementing_the_agent_metadata_service.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
--- | ||
jupytext: | ||
formats: md:myst | ||
text_representation: | ||
extension: .md | ||
format_name: myst | ||
--- | ||
|
||
(implementing_the_agent_metadata_service)= | ||
# Implementing the agent metadata service | ||
|
||
## About the agent metadata service | ||
|
||
Before FlytePropeller sends a request to the agent server, it needs to know four things: | ||
|
||
- The name of the agent | ||
- Which task category the agent supports | ||
- The version of the task category | ||
- Whether the agent executes tasks synchronously or asynchronously | ||
|
||
After FlytePropeller obtains this metadata, it can send a request to the agent deployment using the correct gRPC method. | ||
|
||
:::{note} | ||
- An agent can support multiple task categories. | ||
- We will use the combination of [task category][version] to identify the specific agent's deployment and know whether the task is synchronous or asynchronous in FlytePropeller. | ||
- The task category is `task_type` in flytekit. | ||
::: | ||
|
||
Using the BigQuery Agent as an example: | ||
- The agent's name is `BigQuery Agent`. | ||
- The agent supports `bigquery_query_job_task`. | ||
- The agent's version is `0`. | ||
- By default, the agent executes tasks asynchronously. | ||
|
||
## Implement the agent metadata service | ||
|
||
To implement the agent metadata service, you must do two things: | ||
|
||
1. Implement the agent metadata service. | ||
2. Add the agent metadata service to the agent server. | ||
|
||
You can refer to [base_agent.py](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/base_agent.py), [agent_service.py](https://github.com/flyteorg/flytekit/blob/master/flytekit/extend/backend/agent_service.py), and [serve.py](https://github.com/flyteorg/flytekit/blob/master/flytekit/clis/sdk_in_container/serve.py) to see how the agent metadata service is implemented in flytekit's agent server. | ||
|
||
Those gRPC methods are generated by [flyteidl](https://github.com/flyteorg/flyte/blob/master/flyteidl/protos/flyteidl/service/agent.proto) and you can import them from [here](https://github.com/flyteorg/flyte/tree/master/flyteidl/gen). | ||
|
||
:::{note} | ||
You can search the keyword `metadata` to find implementations in those files. | ||
::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
94 changes: 94 additions & 0 deletions
94
docs/flyte_agents/testing_agents_in_a_local_development_cluster.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
--- | ||
jupytext: | ||
formats: md:myst | ||
text_representation: | ||
extension: .md | ||
format_name: myst | ||
--- | ||
|
||
(testing_agents_in_a_local_development_cluster)= | ||
# Testing agents in a local development cluster | ||
|
||
The Flyte agent service runs in a separate deployment instead of inside FlytePropeller. To test an agent server in a local development cluster, you must run both the single binary and agent server at the same time, allowing FlytePropeller to send requests to your local agent server. | ||
|
||
## Backend plugin vs agent service architecture | ||
|
||
To understand why you must run both the single binary and agent server at the same time, it is helpful to compare the backend plugin architecture to the agent service architecture. | ||
|
||
### Backend plugin architecture | ||
|
||
In this architecture, FlytePropeller sends requests through the SDK: | ||
|
||
![image.png](https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/concepts/agents/plugin_life_cycle.png) | ||
|
||
### Agent service architecture | ||
|
||
With the agent service framework: | ||
1. Flyteplugins send gRPC requests to the agent server. | ||
2. The agent server sends requests through the SDK and returns the query data. | ||
|
||
![image.png](https://raw.githubusercontent.com/flyteorg/static-resources/main/flyte/concepts/agents/async_agent_life_cycle.png) | ||
|
||
## Configuring the agent service in development mode | ||
|
||
1. Start the demo cluster in dev mode: | ||
```bash | ||
flytectl demo start --dev | ||
``` | ||
|
||
2. Start the agent grpc server: | ||
```bash | ||
pyflyte serve agent | ||
``` | ||
|
||
3. Update the config for the task handled by the agent in the single binary yaml file. | ||
```bash | ||
cd flyte | ||
vim ./flyte-single-binary-local.yaml | ||
``` | ||
|
||
```yaml | ||
:emphasize-lines: 9 | ||
tasks: | ||
task-plugins: | ||
enabled-plugins: | ||
- agent-service | ||
- container | ||
- sidecar | ||
- K8S-ARRAY | ||
default-for-task-types: | ||
- bigquery_query_job_task: agent-service | ||
- container: container | ||
- container_array: K8S-ARRAY | ||
``` | ||
```yaml | ||
plugins: | ||
# Registered Task Types | ||
agent-service: | ||
defaultAgent: | ||
endpoint: "localhost:8000" # your grpc agent server port | ||
insecure: true | ||
timeouts: | ||
GetTask: 10s | ||
defaultTimeout: 10s | ||
``` | ||
4. Start the Flyte server with the single binary config file: | ||
```bash | ||
make compile | ||
./flyte start --config ./flyte-single-binary-local.yaml | ||
``` | ||
|
||
5. Set up your secrets: | ||
In the development environment, you can set up your secrets on your local machine by adding secrets to `/etc/secrets/SECRET_NAME`. | ||
|
||
Since your agent server is running locally rather than within Kubernetes, it can retrieve the secret from your local file system. | ||
|
||
6. Test your agent task: | ||
```bash | ||
pyflyte run --remote agent_workflow.py agent_task | ||
``` | ||
|
||
:::{note} | ||
You must build an image that includes the plugin for the task and specify its config with the [`--image` flag](https://docs.flyte.org/en/latest/api/flytekit/pyflyte.html#cmdoption-pyflyte-run-i) when running `pyflyte run` or in an {ref}`ImageSpec <imagespec>` definition in your workflow file. | ||
::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters