Skip to content

Commit

Permalink
update kafka proxy conent and sidebar
Browse files Browse the repository at this point in the history
  • Loading branch information
vordimous committed Sep 21, 2023
1 parent 08799b5 commit 2d1b18e
Show file tree
Hide file tree
Showing 8 changed files with 51 additions and 61 deletions.
12 changes: 6 additions & 6 deletions src/.vuepress/sidebar/en.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,11 @@ export const enSidebar = sidebar({
],
},
{
text: "Apache Kafka Proxying",
text: "Kafka Proxying",
link: "concepts/kafka-proxies/rest-proxy.md",
children: [
{
text: "REST-Kafka Proxy",
text: "REST Kafka Proxy",
collapsible: true,
link: "concepts/kafka-proxies/rest-proxy.md",
children: [
Expand Down Expand Up @@ -152,7 +152,7 @@ export const enSidebar = sidebar({
],
},
{
text: "SSE-Kafka Proxy",
text: "SSE Kafka Proxy",
collapsible: true,
link: "concepts/kafka-proxies/sse-proxy.md",
children: [
Expand Down Expand Up @@ -181,7 +181,7 @@ export const enSidebar = sidebar({
],
},
{
text: "gRPC-Kafka Proxy",
text: "gRPC Kafka Proxy",
collapsible: true,
link: "concepts/kafka-proxies/grpc-proxy.md",
children: [
Expand All @@ -196,7 +196,7 @@ export const enSidebar = sidebar({
],
},
{
text: "MQTT-Kafka Proxy",
text: "MQTT Kafka Proxy",
collapsible: true,
link: "concepts/kafka-proxies/mqtt-proxy.md",
children: [
Expand All @@ -205,7 +205,7 @@ export const enSidebar = sidebar({
link: "concepts/kafka-proxies/mqtt-proxy.md",
},
{
text: "Create a Simple REST API",
text: "Create a Simple MQTT Broker",
link: "tutorials/mqtt/mqtt-intro.md",
}
],
Expand Down
2 changes: 1 addition & 1 deletion src/concepts/kafka-proxies/grpc-proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
description: This guide will walk through each unique gRPC message request and response design and how Zilla is configured to manage the connection for each.
---

# gRPC Proxy
# gRPC Kafka Proxy

This guide will walk through each unique gRPC message request and response design and how Zilla is configured to manage the connection for each.

Expand Down
74 changes: 31 additions & 43 deletions src/concepts/kafka-proxies/mqtt-proxy.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
---
description: This guide will walk through each unique way Zilla manages MQTT Pub/Sub connections and messages.
description: This guide will walk through the way Zilla manages MQTT Pub/Sub connections and messages.
---

# MQTT Proxy
# MQTT Kafka Proxy

This guide will walk through each unique way Zilla manages MQTT Pub/Sub connections and messages.
This guide will walk through the way Zilla manages MQTT Pub/Sub connections and messages.

An MQTT server typically acts as a broker between publishers and subscribers. This requires a complex protocol to manage the wide range of small IoT device use cases. By proxying these messages on and off of Kafka with the [mqtt-kafka](../../reference/config/bindings/binding-mqtt-kafka.md) binding, IoT devices can transmit data to a wider range of tech stacks and adapting to business needs.
An MQTT server acts as a broker between publishers and subscribers. This requires a complex protocol to manage the wide range of IoT devices and use cases. By proxying these messages on and off of Kafka with the [mqtt-kafka](../../reference/config/bindings/binding-mqtt-kafka.md) binding, IoT devices can transmit data to a wider range of tech stacks, adapting to more business needs.

Unlike other proxies, Zilla manages the different MQTT topics instead of passing them down to kafka. This allows more control over the Kafka architecture. Subscribers and publishers won't need to do any extra work since Zilla supports the native MQTT protocol.


> MQTT reflect kinda like echo but the relay is from all clients and not a 1:1. Pub/Sub implies 1-many pub and 1-many sub.
Unlike other proxies, Zilla manages the different MQTT topics instead of blindly passing them down to Kafka. This way the Kafka architecture can be optimized for MQTT Pub/Sub. MQTT client subscribers and publishers will communicate with Zilla the same as any broker.

## Step 1: Declaring the broker

Expand All @@ -26,65 +23,56 @@ mqtt_server:
mqtt_kafka_proxy:
type: mqtt-kafka
kind: proxy
options:
topics:
sessions: mqtt-sessions
messages: mqtt-messages
retained: mqtt-retained
```
### Protocol version
MQTT v5, no clean_session.
The Zilla MQTT `server` supports the [MQTT v5.0 Specification].

::: info Feature Coming Soon <HopeIcon icon="circle-right"/>
MQTT v3.1 and v3.1.1 support is currently on the [Zilla roadmap](https://github.com/orgs/aklivity/projects/4). Star and watch the [Zilla repo](https://github.com/aklivity/zilla/releases) for new releases!
[MQTT v3.1.1 Specification] support is currently on the [Zilla roadmap]. Star and watch the [Zilla repo] for new releases!
:::

[MQTT v5.0 Specification]:https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html
[MQTT v3.1.1 Specification]:http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html
[Zilla roadmap]:https://github.com/orgs/aklivity/projects/4
[Zilla repo]:https://github.com/aklivity/zilla/releases

### QOS

At most once (QoS 0)
At least once (QoS 1)
Exactly once (QoS 2)
The Zilla MQTT `server` supports the "At most once (QoS 0)" Quality of Service flag.

::: info Feature Coming Soon <HopeIcon icon="circle-right"/>
At least once (QoS 1) and Exactly once (QoS 2) delivery will be support is currently on the [Zilla roadmap](https://github.com/orgs/aklivity/projects/4). Star and watch the [Zilla repo](https://github.com/aklivity/zilla/releases) for new releases!
At least once (QoS 1) and Exactly once (QoS 2) delivery will be support is currently on the [Zilla roadmap]. Star and watch the [Zilla repo] for new releases!
:::

## Step 2: Pub/Sub message reflect with Kafka

Zilla manages MQTT pub/sub using three kafka topics. The specific topic names can be configured using the [options.topics](../../reference/config/bindings/binding-mqtt-kafka.md#options-topics) property
Zilla manages MQTT pub/sub using three kafka topics. The specific topic names can be configured using the [options.topics](../../reference/config/bindings/binding-mqtt-kafka.md#options-topics) property.

```yaml{7-8}
mqtt_kafka_proxy:
type: mqtt-kafka
kind: proxy
options:
topics:
sessions: mqtt-sessions
messages: mqtt-messages
retained: mqtt-retained
```yaml
topics:
messages: mqtt-messages
retained: mqtt-retained
sessions: mqtt-sessions
```

The message topic becomes the kafka key
all messages on the messages topic
When the `retain` flag is set, a copy of the message is stored on the `retained` topic
### Messages on Kafka

All MQTT messages brokered by Zilla are published on the `messages` Kafka topic. The MQTT message topic becomes the kafka key

## Step 3: Client Session Management
### Retaining Messages

Now that messages are in Kafka we need to send them to the gRPC services responsible for processing them. For this, we will be using the [mqtt-kafka](../../reference/config/bindings/binding-mqtt-kafka.md) binding.

```yaml{6}
mqtt_kafka_proxy:
type: mqtt-kafka
kind: proxy
options:
topics:
sessions: mqtt-sessions
messages: mqtt-messages
retained: mqtt-retained
```
MQTT messages with the `retain` flag set will have a copy published on the `retained` Kafka topic.

### Sessions
### Session Management

MQTT connect and disconnect messages are published on the `sessions` Kafka topic.

## Step 3: Authorizing clients

Expand Down
6 changes: 5 additions & 1 deletion src/concepts/kafka-proxies/rest-proxy.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
# REST Proxy
---
description: Zilla lets you configure application-centric REST API endpoints that unlock Kafka event-driven architectures.
---

# REST Kafka Proxy
<!-- TODO enable -->
<!-- markdownlint-disable -->

Expand Down
2 changes: 1 addition & 1 deletion src/concepts/kafka-proxies/sse-proxy.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SSE Proxy
# SSE Kafka Proxy

There is an increasing rise in integrating the event stream into front ends where companies are starting to adopt Server-sent Events (SSE) standards. `SSE` naturally fits into the event-driven architecture and you will be able to take advantage of all the benefits it provides such as SDK-free and the ability to auto-reconnect in case of an unstable connection(Be resilient to faults). Zilla supports SSE protocol that you can easily configure the frontend SSE with Kafka topic.

Expand Down
6 changes: 2 additions & 4 deletions src/how-tos/amazon-msk/public-proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ description: Securely access your Amazon MSK cluster via the internet.
::: tip Estimated time to complete 20-30 minutes.
:::

## Overview

The [Zilla Plus (Public MSK Proxy)](https://aws.amazon.com/marketplace/pp/prodview-jshnzslazfm44) lets authorized Kafka clients connect, publish messages and subscribe to topics in your Amazon MSK cluster via the internet.

By automating the configuration of an internet-facing network load balancer and auto-scaling group of stateless proxies to access your MSK cluster via the public internet, Kafka clients can connect, publish messages and subscribe to topics in your Amazon MSK cluster from outside AWS.
Expand All @@ -23,11 +21,11 @@ You will need to choose a wildcard DNS pattern to use for public internet access

Both `Development` and `Production` deployment options are available.

### Development
## Development

Follow the [Development](./development.md) guide to setup connectivity to your MSK cluster from your local development environment via the internet using a locally trusted TLS server certificate for the example wildcard DNS pattern `*.aklivity.example.com`.

### Production
## Production

Follow the [Production](./production.md) guide to setup connectivity to your MSK cluster from anywhere on the internet using a globally trusted TLS server certificate for a wildcard DNS pattern under your control. We use `*.example.aklivity.io` to illustrate the steps.

Expand Down
4 changes: 2 additions & 2 deletions src/reference/config/bindings/binding-mqtt.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,9 @@ mqtt_server:
## Summary
Defines a binding with `mqtt 5.0` protocol support, with `server` behavior.
Defines a binding with [MQTT v5.0](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html) protocol support, with `server` behavior.

The `server` kind `mqtt` binding decodes `mqtt 5.0` protocol on the inbound network stream, producing higher level application streams for each `publish` or `subscribe` `topic`. The `session` state is also described by a higher level application stream.
The `server` kind `mqtt` binding decodes the MQTT protocol on the inbound network stream, producing higher level application streams for each `publish` or `subscribe` `topic`. The `session` state is also described by a higher level application stream.

Conditional routes based on the `topic` `name` are used to route these application streams to an `exit` binding.

Expand Down
6 changes: 3 additions & 3 deletions src/tutorials/mqtt/mqtt-intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Create these files, `zilla.yaml` and `docker-compose.yaml`, in the same director

@tab zilla.yaml

```yaml {10,23-25,37,38}
```yaml {11,26-28,41-42}
name: MQTT-intro
bindings:

Expand Down Expand Up @@ -81,7 +81,7 @@ bindings:
@tab docker-compose.yaml
```yaml
```yaml {9,40-42}
version: '3'
services:

Expand Down Expand Up @@ -175,7 +175,7 @@ Go deeper into this concept with the [mqtt.kafka.reflect](https://github.com/akl

## Going Deeper

Try out more HTTP examples:
Try out more MQTT examples:

- [mqtt.kafka.reflect](https://github.com/aklivity/zilla-examples/tree/main/mqtt.kafka.reflect)
- [mqtt.kafka.reflect.jwt](https://github.com/aklivity/zilla-examples/tree/main/mqtt.kafka.reflect.jwt)

0 comments on commit 2d1b18e

Please sign in to comment.