Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: tutorial for managing schemas with Schema Registry #2331

Merged
merged 67 commits into from
Mar 16, 2024
Merged
Changes from 64 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
f356e1f
initial structure
Arya-Gupta Nov 18, 2023
7db40f4
added metadata
Arya-Gupta Nov 18, 2023
90e3262
added introduction
Arya-Gupta Nov 18, 2023
c0f04c7
changed page weight
Arya-Gupta Nov 18, 2023
dd81c6d
added background context
Arya-Gupta Nov 18, 2023
c7eac29
added prerequisites
Arya-Gupta Nov 18, 2023
5f4744c
added temp files for preview
Arya-Gupta Nov 19, 2023
c6dd3cf
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Nov 27, 2023
e0a693c
Update pages/docs/tutorials/kafka/managing-schemas-using-schema-regis…
Arya-Gupta Nov 27, 2023
4b16656
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Dec 6, 2023
07f8df5
implemented feedback
Arya-Gupta Dec 6, 2023
02a622b
added steps
Arya-Gupta Dec 6, 2023
b3a74d4
added installation steps
Arya-Gupta Dec 6, 2023
fe4f735
removed mentions of we according to the style guide
Arya-Gupta Dec 6, 2023
e35d18d
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Dec 21, 2023
4aa9887
Update managing-schemas-using-schema-registry.md
Arya-Gupta Dec 21, 2023
59e4bb0
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 10, 2024
8111f1d
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 11, 2024
398e6bd
added curl command for creating avro schema artifact
Arya-Gupta Jan 11, 2024
f5f3970
added usage of
Arya-Gupta Jan 11, 2024
719c97a
added usage of json pointer
Arya-Gupta Jan 11, 2024
56f08db
Merge branch 'managing-schemas' of https://github.com/Arya-Gupta/webs…
Arya-Gupta Jan 11, 2024
83b2f33
specified apicurio registry version in command
Arya-Gupta Jan 11, 2024
9da1a66
Update pages/docs/tutorials/kafka/managing-schemas-using-schema-regis…
Arya-Gupta Jan 16, 2024
0230f93
Update pages/docs/tutorials/kafka/managing-schemas-using-schema-regis…
Arya-Gupta Jan 16, 2024
8794a71
added link to apicurio registry official docs
Arya-Gupta Jan 16, 2024
a126717
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 18, 2024
54b67f0
updated docker command
Arya-Gupta Jan 18, 2024
286e33c
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 21, 2024
0aa96ea
added url for downloading avro schema
Arya-Gupta Jan 21, 2024
e9b4711
Merge branch 'managing-schemas' of https://github.com/Arya-Gupta/webs…
Arya-Gupta Jan 21, 2024
f728a1d
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 23, 2024
ab52ef6
added prerequisites section
Arya-Gupta Jan 23, 2024
40c6484
changed optional step to remember block
Arya-Gupta Jan 23, 2024
55a59a3
added summary
Arya-Gupta Jan 23, 2024
d2f6211
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 26, 2024
f78f7c7
fixed link to previous tutorial
Arya-Gupta Jan 26, 2024
b826115
added code snippet
Arya-Gupta Jan 26, 2024
41907b5
added next steps
Arya-Gupta Jan 27, 2024
fa1ccb0
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 29, 2024
369f8a2
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Jan 30, 2024
aba245b
changed description
Arya-Gupta Jan 30, 2024
1d5faab
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Feb 5, 2024
37200e2
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Feb 7, 2024
fb0a1fa
Added heading to each step
Arya-Gupta Feb 7, 2024
58f11ef
fixed anchor text
Arya-Gupta Feb 7, 2024
704dab5
added url
Arya-Gupta Feb 7, 2024
d9c3a03
tw editorial review
quetzalliwrites Feb 9, 2024
38c96dc
Delete pages/docs/tutorials/kafka/_section.md
quetzalliwrites Feb 9, 2024
ebcfb8e
Delete pages/docs/tutorials/kafka/index.md
quetzalliwrites Feb 9, 2024
39926d8
Merge branch 'master' into managing-schemas
quetzalliwrites Feb 9, 2024
9209ae0
Merge branch 'master' into managing-schemas
quetzalliwrites Feb 10, 2024
538d531
fix page weight
quetzalliwrites Feb 10, 2024
00006c2
Merge branch 'master' into managing-schemas
quetzalliwrites Feb 12, 2024
068a506
Merge branch 'master' into managing-schemas
quetzalliwrites Feb 14, 2024
d426e86
modified command for starting apicurio registry
Arya-Gupta Feb 16, 2024
3a9d583
Merge branch 'master' into managing-schemas
quetzalliwrites Feb 16, 2024
c75bb44
updated heading
Arya-Gupta Feb 25, 2024
bd2c3b4
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Feb 25, 2024
534572d
Merge branch 'master' into managing-schemas
quetzalliwrites Feb 28, 2024
ca1a554
updated introduction to mention previous tutorial
Arya-Gupta Mar 3, 2024
db82383
added AsyncAPI document with avro schema
Arya-Gupta Mar 3, 2024
b83b0a9
Merge branch 'asyncapi:master' into managing-schemas
Arya-Gupta Mar 3, 2024
04bc8b1
grammar fixes
quetzalliwrites Mar 15, 2024
c4e35f9
grammar fixes
quetzalliwrites Mar 15, 2024
00614f2
Merge branch 'master' into managing-schemas
quetzalliwrites Mar 15, 2024
ad78b4e
fix link
quetzalliwrites Mar 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions pages/docs/tutorials/kafka/managing-schemas-using-schema-registry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
title: Managing schemas using Schema Registry
description: A tutorial teaching how to manage schemas using Schema Registry.
weight: 250
---

## Introduction
In the [previous Kafka and Avro configuration tutorial](/docs/tutorials/kafka/configure-kafka-avro), you learned how to add Apache Avro schemas to your AsyncAPI document. Now, you will learn how to save your schema in a central Schema Registry and reuse it.

## Background context
The need for schema management has become increasingly prevalent to handle the evolving complexity of modern Event-Driven Architecture. A Schema Registry is a centralized service that stores and maintains schemas for data exchanged between various components of a modern distributed system. Validating exchanged data maintains data consistency and compatibility.

While several Schema Registry implementations exist, you will use the [Apicurio Registry](https://www.apicur.io/registry/) for this tutorial. Apicurio Registry is a popular open-source Schema Registry implementation that supports multiple serialization formats and facilitates schema management for diverse data in distributed systems. You will use Apicurio Registry combined with Avro, a language-neutral data serialization system.

## Prerequisites
[Install Docker](https://docs.docker.com/engine/install/) from the official website.

Arya-Gupta marked this conversation as resolved.
Show resolved Hide resolved
## AsyncAPI document with Avro Schema
The previous tutorial taught you how to write an AsyncAPI document for Kafka messages using the Avro schema. Here's an example of what an AsyncAPI document fully equipped with Avro schema looks like:
```
asyncapi: 3.0.0
info:
title: User Signup API
version: 1.0.0
description: The API notifies you whenever a new user signs up in the application.
servers:
kafkaServer:
host: test.mykafkacluster.org:8092
description: Kafka Server
protocol: kafka
operations:
onUserSignedUp:
action: receive
channel:
$ref: '#/channels/userSignedUp'
channels:
userSignedUp:
description: This channel contains a message per each user who signs up in our application.
address: user_signedup
messages:
userSignedUp:
$ref: '#/components/messages/userSignedUp'
components:
messages:
userSignedUp:
payload:
schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
schema:
type: record
name: UserSignedUp
namespace: com.company
doc: User sign-up information
fields:
- name: userId
type: int
- name: userEmail
type: string
```

### Start Apicurio Registry
Start the Apicurio Registry locally with the following docker command:

```
docker run --env CORS_ALLOWED_ORIGINS='*' -it -p 8080:8080 apicurio/apicurio-registry-mem:2.5.8.Final
```

### Upload Avro Schema
quetzalliwrites marked this conversation as resolved.
Show resolved Hide resolved
Once your local instance of Apicurio Registry is running, upload your Avro schema. Open a new terminal window and create an Avro schema artifact with the following command:

```
curl \
http://localhost:8080/apis/registry/v2/groups/my-group/artifacts \
-X POST \
-H "Content-Type: application/json; artifactType=AVRO" \
-H "X-Registry-ArtifactId: UserSignedUp" \
--data @- << EOF
{
"type": "record",
"name": "UserSignedUp",
"namespace": "com.company",
"doc": "User sign-up information",
"fields": [
{
"name": "userId",
"type": "int"
},
{
"name": "userEmail",
"type": "string"
}
]
}
EOF
```

<Remember>
Download your Avro schema by visiting the following URL:
http://localhost:8080/apis/registry/v2/groups/my-group/artifacts/UserSignedUp.
</Remember>

### Update schema reference
One alternative is to keep your schema in a separate file, as you learned in the previous tutorial, [describe Kafka message payload using Avro schema](/docs/tutorials/kafka/configure-kafka-avro). After uploading your Avro schema, remove the schema from your AsyncAPI document and add a `$ref` pointing to the previous step's URL.
```
$ref: http://localhost:8080/apis/registry/v2/groups/my-group/artifacts/UserSignedUp
```

```
asyncapi: 3.0.0
info:
title: User Signup API
version: 1.0.0
description: The API notifies you whenever a new user signs up in the application.
servers:
kafkaServer:
host: test.mykafkacluster.org:8092
description: Kafka Server
protocol: kafka
operations:
onUserSignedUp:
action: receive
channel:
$ref: '#/channels/userSignedUp'
channels:
userSignedUp:
description: This channel contains a message per each user who signs up in our application.
address: user_signedup
messages:
userSignedUp:
$ref: '#/components/messages/userSignedUp'
components:
messages:
userSignedUp:
payload:
schemaFormat: 'application/vnd.apache.avro+json;version=1.9.0'
schema:
$ref: http://localhost:8080/apis/registry/v2/groups/my-group/artifacts/UserSignedUp
```
quetzalliwrites marked this conversation as resolved.
Show resolved Hide resolved

Arya-Gupta marked this conversation as resolved.
Show resolved Hide resolved
## Summary
In this tutorial, you managed Avro schemas using a centralized schema registry that enables you to share schemas across multiple applications. The good news is that this approach is valid for various other schema types!

Arya-Gupta marked this conversation as resolved.
Show resolved Hide resolved
## Next steps
Now that you have learned how to manage schemas, check out the [bindings with Kafka tutorial](/docs/tutorials/kafka/bindings-with-Kafka) to start sending messages between your services.
quetzalliwrites marked this conversation as resolved.
Show resolved Hide resolved
Loading