Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add a Why section to the readme #91

Merged
merged 2 commits into from
May 28, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,43 @@
# Maestro

Maestro is a system to leverage [CloudEvents](https://cloudevents.io/) to transport Kubernetes resources to the target clusters, and then transport the resource status back. The resources are stored in a database and the status is updated in the database as well. The system is composed of two parts: the Maestro server and the Maestro agent.
- The Maestro server is responsible for storing the resources and their status. And sending the resources to the message broker in the CloudEvents format. The Maestro server provides Resful APIs and gRPC APIs to manage the resources.
- Maestro agent is responsible for receiving the resources and applying them to the target clusters. And reporting back the status of the resources.

## Architecture

Taking MQTT as an example:

![Maestro Architecture](./arch.png)

## Why

Maestro was created to address several critical challenges associated with managing Kubernetes clusters at scale.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph is good in that we should see performance and scalability improvements by using Maestro, as you note.

Another motivation (I think, perhaps, primary motivation) was to improve the security architecture.

By using Maestro, we do not need kubeconfigs. Our Cluster Service (CS) deployment would no longer need kubeconfigs or direct access to customer clusters. CS just uses the Maestro API.

Only the Maestro consumer in a management cluster needs kubeconfigs, thus improving security.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Mark for the review!

Could you please help me word it in a way that's more generic and not mention CS, SD, MC, HCP etc?
I am trying to show maestro as a possible solution for another project and it's not related to SD. Maybe closer to self managed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a generic sentence about not needing kubeconfig. Maybe it's a bit vague but PTAL. Thanks.

Traditional Kubernetes native and custom resources rely heavily on etcd,
which has well-known limitations in terms of scalability and performance.
As the number of clusters increases, the strain on etcd can lead to significant issues,
making it difficult to achieve high-scale Kubernetes cluster management.
To overcome these challenges, Maestro introduces a system that leverages a traditional relational database for storing relevant data,
providing a more scalable and efficient solution.

### Motivations

**Scalability**: Kubernetes-based solutions often face scalability issues due to the limitations of etcd.
Maestro aims to overcome these issues by decoupling resource management from the etcd store,
allowing for more efficient handling of a large number of clusters.
This approach supports the goal of managing up to 200,000+ clusters without linear scaling of the infrastructure.

**Cost Effectiveness**: Running a central orchestrator in a Kubernetes-based environment can be costly.
Maestro reduces infrastructure and maintenance costs by leveraging a relational database and optimized content delivery mechanisms.

**Improve Feedback Loop**: Traditional Kubernetes solutions have limitations in the feedback loop,
making it difficult to observe the state of resources effectively.
Maestro addresses this by providing a robust feedback loop mechanism,
ensuring that resource status is continuously updated and monitored.

**Improve Security Architecture**: Maestro enhances security by eliminating the need for kubeconfigs,
reducing the need for direct access to clusters.

## Run for the first time

### Make a build, run postgres and mqtt broker
Expand Down
Loading