-
Notifications
You must be signed in to change notification settings - Fork 61
Add multicloud federated learning parttern #570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
5aa4742
d9c2d35
2a0966e
976f151
144cd7b
385d606
904653f
0a75f38
82b4a1b
e96714c
f86ef9c
744d3a8
e40cadf
ac50e61
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,79 @@ | ||||||||||
--- | ||||||||||
title: Multicloud Federated Learning | ||||||||||
date: 2025-05-23 | ||||||||||
summary: This pattern helps you develop and deploy federated learning applications on an open hybrid cloud via Open Cluster Management. | ||||||||||
rh_products: | ||||||||||
- Red Hat Advanced Cluster Management | ||||||||||
- Red Hat OpenShift Container Platform | ||||||||||
industries: | ||||||||||
- General | ||||||||||
aliases: /multicloud-federated-learning/ | ||||||||||
links: | ||||||||||
install: getting-started | ||||||||||
feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform | ||||||||||
bugs: https://github.com/open-cluster-management-io/addon-contrib/issues | ||||||||||
ci: multicloudfederatedlearning | ||||||||||
--- | ||||||||||
|
||||||||||
:toc: | ||||||||||
:imagesdir: /images | ||||||||||
:_content-type: ASSEMBLY | ||||||||||
include::modules/comm-attributes.adoc[] | ||||||||||
|
||||||||||
== Multicloud Federated Learning | ||||||||||
|
||||||||||
=== Background | ||||||||||
|
||||||||||
As machine learning (ML) evolves, protecting data privacy becomes increasingly important. Since ML depends on large volumes of data, it is essential to secure that data without disrupting the learning process. | ||||||||||
|
||||||||||
Federated learning addresses this by allowing multiple clusters or organizations to collaboratively train models without sharing sensitive data. Computation happens where the data lives, ensuring privacy, regulatory compliance, and efficiency. | ||||||||||
|
||||||||||
By integrating federated learning with {rh-rhacm-first}, this pattern provides an automated and scalable solution for deploying federated learning workloads across hybrid and multicluster environments. | ||||||||||
|
||||||||||
==== Technologies | ||||||||||
* Open Cluster Management (OCM) | ||||||||||
- ManagedCluster | ||||||||||
- ManifestWork | ||||||||||
- Placement | ||||||||||
... | ||||||||||
* Federated Learning Frameworks | ||||||||||
- Flower | ||||||||||
- OpenFL | ||||||||||
... | ||||||||||
* Grafana | ||||||||||
* OpenTelemetry | ||||||||||
|
||||||||||
=== Why Use Advanced Cluster Management for Federated Learning? | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Suggested change
|
||||||||||
|
||||||||||
- **Advanced Cluster Management (ACM)** simplifies and automates the deployment and orchestration of federated learning workloads across clusters: | ||||||||||
|
||||||||||
- **Automatic Deployment & Simplified Operations**: ACM provides a unified and automated approach to running federated learning workflows across different runtimes (e.g., Flower, OpenFL). Its controller manages the entire federated learning lifecycle—including setup, coordination, status tracking, and teardown—across multiple clusters in a multicloud environment. This eliminates repetitive manual configurations, significantly reduces operational overhead, and ensures consistent, scalable federated learning deployments. | ||||||||||
|
||||||||||
- **Dynamic Client Selection**: ACM is scheduling capabilities allow federated learning clients to be selected not only based on where the data resides, but also dynamically based on cluster labels, resource availability, and governance criteria. This enables a more adaptive and intelligent approach to client participation. | ||||||||||
|
||||||||||
Together, these capabilities support a **flexible federated learning client model**, where clusters can join or exit the training process dynamically, without requiring static or manual configuration. | ||||||||||
|
||||||||||
=== Benefits | ||||||||||
|
||||||||||
- Privacy-preserving training without moving sensitive data | ||||||||||
|
||||||||||
- Automated dynamic federated learning client orchestration across distributed clusters | ||||||||||
|
||||||||||
- Adaptable to different federated learning frameworks, such as OpenFL and Flower | ||||||||||
|
||||||||||
- Scalability across hybrid and edge clusters | ||||||||||
|
||||||||||
- Lower infrastructure and operational costs | ||||||||||
|
||||||||||
This approach empowers organizations to build smarter, privacy-first AI solutions with less complexity and more flexibility. | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Global comment: Avoid anthropomorphism
Suggested change
|
||||||||||
|
||||||||||
=== Architecture | ||||||||||
|
||||||||||
image::/images/multicloud-federated-learning/multicluster-federated-learning-workflow.png[multicloud-federated-learning-workflow] | ||||||||||
|
||||||||||
- In this architecture, a central **Hub Cluster** acts as the aggregator, running the Federated Learning controller and scheduling workloads using ACM APIs like `Placement` and `ManifestWork`. | ||||||||||
|
||||||||||
- Multiple **Managed Clusters**, potentially across different clouds, serve as federated learning clients—each holding private data. These clusters pull the global model from the hub, train it locally, and push model updates back. | ||||||||||
|
||||||||||
- The controller manages this lifecycle using custom resources and supports runtimes like Flower and OpenFL. This setup enables scalable, multi-cloud model training with **data privacy preserved by design**, requiring no changes to existing federated learning training code. | ||||||||||
|
Original file line number | Diff line number | Diff line change | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,233 @@ | ||||||||||||
--- | ||||||||||||
title: Getting Started | ||||||||||||
weight: 10 | ||||||||||||
aliases: /multicloud-federated-learning/getting-started/ | ||||||||||||
--- | ||||||||||||
|
||||||||||||
:toc: | ||||||||||||
:imagesdir: /images | ||||||||||||
:_content-type: ASSEMBLY | ||||||||||||
include::modules/comm-attributes.adoc[] | ||||||||||||
|
||||||||||||
== Deploying the Multicloud Federated Learning Pattern | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
=== Prerequisites | ||||||||||||
|
||||||||||||
==== Ensure the following tools are installed: | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
- link:https://kubernetes.io/docs/reference/kubectl/[`kubectl`] | ||||||||||||
- link:https://kubectl.docs.kubernetes.io/installation/kustomize/[`kustomize`] | ||||||||||||
- link:https://kind.sigs.k8s.io/[`kind`] (recommended version > v0.9.0) | ||||||||||||
- link:https://www.gnu.org/software/make/[`make`] (for build automation) | ||||||||||||
|
||||||||||||
==== Optional (for container image building): | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
- link:https://podman.io/[Podman] or link:https://www.docker.com/[Docker] | ||||||||||||
- link:https://go.dev/doc/install[Go] (version 1.19 or higher) | ||||||||||||
|
||||||||||||
===== Advanced Cluster Management Environment | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
Prepare at least three clusters: one hub cluster and two managed clusters. | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
Verify the managed clusters are registered on the hub by running: | ||||||||||||
|
||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl get mcl | ||||||||||||
---- | ||||||||||||
|
||||||||||||
Example output: | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE | ||||||||||||
cluster1 true https://api.***.com:6443 True True 5m | ||||||||||||
cluster2 true https://api.***.com:6443 True True 5m | ||||||||||||
---- | ||||||||||||
|
||||||||||||
=== Deploy Federated Learning Controller | ||||||||||||
|
||||||||||||
. Clone and navigate to the repository: | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ git clone [email protected]:open-cluster-management-io/addon-contrib.git | ||||||||||||
$ cd federated-learning-controller | ||||||||||||
---- | ||||||||||||
|
||||||||||||
. Build and push the controller image (or use pre-built `quay.io/myan/federated-learning-controller:latest`): | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ make docker-build docker-push IMG=<IMG> | ||||||||||||
---- | ||||||||||||
|
||||||||||||
. Deploy the controller to the hub cluster: | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl config use-context kind-hub | ||||||||||||
$ make deploy IMG=<controller-image> NAMESPACE=<controller-namespace(default is open-cluster-management)> | ||||||||||||
---- | ||||||||||||
|
||||||||||||
. Verify the deployment: | ||||||||||||
+ | ||||||||||||
The federated learning controller is running in the open-cluster-management namespace by default. | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl get pods -n open-cluster-management | ||||||||||||
---- | ||||||||||||
+ | ||||||||||||
Example output | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
NAME READY STATUS RESTARTS AGE | ||||||||||||
cluster-manager-d9db64db5-c7kfj 1/1 Running 0 3d22h | ||||||||||||
cluster-manager-d9db64db5-t7grh 1/1 Running 0 3d22h | ||||||||||||
cluster-manager-d9db64db5-wndd8 1/1 Running 0 3d22h | ||||||||||||
federated-learning-controller-d7df846c9-nb4wc 1/1 Running 0 3d22h | ||||||||||||
---- | ||||||||||||
|
||||||||||||
=== Deploy the Federated Learning Instance | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
|
||||||||||||
. Build the Application Image | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
+ | ||||||||||||
*Note*: You can skip this step by using the pre-built image `quay.io/myan/flower-app-torch:latest`. | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ cd examples/flower | ||||||||||||
|
||||||||||||
$ export REGISTRY=<your-registry> | ||||||||||||
$ export IMAGE_TAG=<your-tag> | ||||||||||||
$ make build-app-image | ||||||||||||
$ make push-app-image | ||||||||||||
---- | ||||||||||||
+ | ||||||||||||
Image format: `<REGISTRY>/flower-app-torch:<IMAGE_TAG>` | ||||||||||||
|
||||||||||||
. Deploy a Federated Learning Instance | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
+ | ||||||||||||
In this example, both the server and clients use the same image—either the one built above or the pre-built `quay.io/myan/flower-app-torch:latest`. Once the resource is created, the server is deployed to the hub cluster, and the clients are prepared for deployment to the managed clusters. | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
+ | ||||||||||||
Create a `FederatedLearning` resource in the controller namespace on the hub cluster: | ||||||||||||
+ | ||||||||||||
[source,yaml] | ||||||||||||
---- | ||||||||||||
apiVersion: federation-ai.open-cluster-management.io/v1alpha1 | ||||||||||||
kind: FederatedLearning | ||||||||||||
metadata: | ||||||||||||
name: federated-learning-sample | ||||||||||||
spec: | ||||||||||||
framework: flower | ||||||||||||
server: | ||||||||||||
image: <REGISTRY>/flower-app-torch:<IMAGE_TAG> | ||||||||||||
rounds: 3 | ||||||||||||
minAvailableClients: 2 | ||||||||||||
listeners: | ||||||||||||
- name: server-listener | ||||||||||||
port: 8080 | ||||||||||||
type: LoadBalancer | ||||||||||||
storage: | ||||||||||||
type: PersistentVolumeClaim | ||||||||||||
name: model-pvc | ||||||||||||
path: /data/models | ||||||||||||
size: 2Gi | ||||||||||||
client: | ||||||||||||
image: <REGISTRY>/flower-app-torch:<IMAGE_TAG> | ||||||||||||
placement: | ||||||||||||
clusterSets: | ||||||||||||
- global | ||||||||||||
predicates: | ||||||||||||
- requiredClusterSelector: | ||||||||||||
claimSelector: | ||||||||||||
matchExpressions: | ||||||||||||
- key: federated-learning-sample.client-data | ||||||||||||
operator: Exists | ||||||||||||
---- | ||||||||||||
|
||||||||||||
. Schedule the Federated Learning Clients into Managed Clusters | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
+ | ||||||||||||
The above configuration schedules only clusters with a `ClusterClaim` having the key `federated-learning-sample.client-data`. You can combine this with other scheduling policies (refer to the Placement API for details). | ||||||||||||
+ | ||||||||||||
Add the `ClusterClaim` object to these clusters own the data for the client: | ||||||||||||
|
||||||||||||
.. **Cluster1: ** | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl config use-context kind-cluster1 | ||||||||||||
|
||||||||||||
$ cat <<EOF | kubectl apply -f - | ||||||||||||
apiVersion: cluster.open-cluster-management.io/v1alpha1 | ||||||||||||
kind: ClusterClaim | ||||||||||||
metadata: | ||||||||||||
name: federated-learning-sample.client-data | ||||||||||||
spec: | ||||||||||||
value: /data/private/cluster1 | ||||||||||||
EOF | ||||||||||||
---- | ||||||||||||
|
||||||||||||
.. **Cluster2: ** | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl config use-context kind-cluster2 | ||||||||||||
$ cat <<EOF | kubectl apply -f - | ||||||||||||
apiVersion: cluster.open-cluster-management.io/v1alpha1 | ||||||||||||
kind: ClusterClaim | ||||||||||||
metadata: | ||||||||||||
name: federated-learning-sample.client-data | ||||||||||||
spec: | ||||||||||||
value: /data/private/cluster2 | ||||||||||||
EOF | ||||||||||||
---- | ||||||||||||
|
||||||||||||
. Check the status of the federated learning instance | ||||||||||||
|
||||||||||||
.. After creating the instance, the server initially shows a status of `Waiting` | ||||||||||||
+ | ||||||||||||
*Example - server in hub cluster:* | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl get pods | ||||||||||||
NAME READY STATUS RESTARTS AGE | ||||||||||||
federated-learning-sample-server-7jnfs 0/1 Completed 0 9m | ||||||||||||
---- | ||||||||||||
|
||||||||||||
.. Once the required clients are ready, status changes to `Running` | ||||||||||||
+ | ||||||||||||
*Example - client in managed cluster:* | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
$ kubectl get pods -n open-cluster-management | ||||||||||||
NAME READY STATUS RESTARTS AGE | ||||||||||||
federated-learning-sample-client-75sc8 0/1 Completed 0 8m | ||||||||||||
---- | ||||||||||||
|
||||||||||||
.. After the training and aggregation rounds complete, the status becomes `Completed` | ||||||||||||
+ | ||||||||||||
*Example - federated learning instance:* | ||||||||||||
+ | ||||||||||||
[source,terminal] | ||||||||||||
---- | ||||||||||||
status: | ||||||||||||
listeners: | ||||||||||||
- address: 172.18.0.2:31166 | ||||||||||||
name: listener(service):federated-learning-sample-server | ||||||||||||
port: 31166 | ||||||||||||
type: NodePort | ||||||||||||
message: Model training successful. Check storage for details | ||||||||||||
phase: Completed | ||||||||||||
---- | ||||||||||||
|
||||||||||||
.. Download and Verify the Trained Model | ||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||
+ | ||||||||||||
The trained MNIST model is saved in the `model-pvc` volume. | ||||||||||||
+ | ||||||||||||
- link:https://github.com/open-cluster-management-io/addon-contrib/blob/main/federated-learning-controller/examples/notebooks/deploy[Deploy a Jupyter notebook server] | ||||||||||||
- link:https://github.com/open-cluster-management-io/addon-contrib/blob/main/federated-learning-controller/examples/notebooks/1.hub-evaluation.ipynb[Validate the model] | ||||||||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if we have CI set up for this. Kindly check this with the Validated Patterns engineering team
cc: @day0hero
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mlabonte-rh @yukinchan - what do we need to do to get this into 'testing' ?