Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal to solve publish/subscribe confusion #618

Closed
Tracked by #520
fmvilas opened this issue Sep 6, 2021 · 99 comments
Closed
Tracked by #520

Proposal to solve publish/subscribe confusion #618

fmvilas opened this issue Sep 6, 2021 · 99 comments
Assignees
Labels
💭 Strawman (RFC 0) RFC Stage 0 (See CONTRIBUTING.md)

Comments

@fmvilas
Copy link
Member

fmvilas commented Sep 6, 2021

TL;DR: What's new?

  • Remote vs Local Servers: Servers got a new kind property. By default, all servers are remote servers (e.g., brokers). However, if the kind property is set to local, that means the application is the one actually exposing the server (e.g., WebSocket or HTTP Server-Sent Events server).
  • Channel Identifiers: Channels are not identified by their "address" anymore. Instead, we define an ID that represents the channel and the "address" is specified separately. This encourages the reuse of channels and eases the process of referencing them and changing their address without breaking the referencing documents.
  • Operations: There's a new top-level object called "operations". It defines the operations (as in actions) this application is performing or going to perform, and in which channel.
  • Send/Receive: We get rid of "publish" and "subscribe" and instead replace them with "send" and "receive". They're both actions an application performs instead of actions an application expects the client to perform. Aside from the confusion "publish" and "subscribe" are creating, some people are incorrectly assuming that AsyncAPI is only meant to describe pub/sub event-driven architectures.
  • Optional channels: from now on, the channels object is optional. The only two fields that are required are asyncapi and info.
  • More reusability: now servers and channels can be defined inside components for reusability purposes. This way, an AsyncAPI document may only contain a components object that can serve as an organization-wide menu of servers, channels, messages, etc.

Abstract

For some years now, we've been discussing how to solve the publish/subscribe perspective problem. Some were arguing we should be giving priority to broker-based architectures because "they are the most common type of architecture AsyncAPI is used for". Although they're not wrong, by simply changing the meaning of publish and subscribe keywords, we'd be leaving many people and use cases out. We have to come up with a way to fix this problem without losing the ability to define client/server interactions too, which are especially useful for public WebSockets and HTTP Server-Sent Events APIs. And more importantly, we need a structure that will allow us to grow well and meet our vision.

The latest Thinking Out Loud episodes with @lornajane, @dalelane, @damaru-inc, and @jmenning-solace have been key to better understand the problem. The intent of this proposal is to be a mix of the ideas and feedback I got from the community. I hope I managed to capture them well 😊

Foundational concepts

We gotta start from the beginning: an AsyncAPI file represents an application. This doesn't change. However, if no channels or operations are provided, the file becomes a "menu", a "library", a collection of reusable items we can later use from other AsyncAPI files.

We have been very focused on the meaning of publish and subscribe but there is another key aspect of the spec that is confusing to many people: servers. As it is right now, the servers keyword holds an object with definitions of the servers (brokers) we may have to connect to. However, it also holds information about the server our application is exposing (HTTP server, WebSocket server, etc.) In some tools, we have been incorrectly assuming that if someone specifies ws as the protocol, it means they want to create a WebSocket server instead of a WebSocket client. But what if someone wants to connect to a broker using the WebSocket protocol? This whole thing about the role of our application has been confusing all of us. As @lornajane pointed out on multiple occasions, an application can be both a server and a client, or just a server, or just a client. Therefore, servers can't be made up of that mix. Exposed server interfaces and remote servers (usually brokers) have to be separated because —even though they look the same— they're semantically different.

Remotes vs Local Servers

This proposal introduces the concept of a remote or local server. Remote servers are those our application has to connect to. They're usually brokers but can also be other kinds of servers.

On the other hand, local servers are server interfaces our application exposes. Their URL defines where clients can reach them.

Example

asyncapi: 3.0.0

servers:
  test:
    url: ws://test.mycompany.com/ws
    protocol: was
    kind: local
    description: The application creates a WebSocket server and listens for messages. Clients can connect on the given URL.
  mosquitto:
    url: mqtt://test.mosquitto.org
    protocol: mqtt
    kind: remote # This is the default value
    description: The application is connecting to the Mosquitto Test broker.

New channels and operations objects

Another common concern related to the current state of publish and subscribe is the channel reusability problem we encounter because these verbs are part of the channel definition. To avoid this problem, we remove the operation verbs from the channel definitions and move them to their own root object operations. Let's have a look at an example:

asyncapi: 3.0.0

channels:
  userSignedUp:
    address: user/signedup
    message:
      payload:
        type: object
        properties:
          email:
            type: string
            format: email

operations:
  sendUserSignedUpEvent:
    channel: userSignedUp
    action: send

There are a few new things here:

  • Channel address: it's the logical address where you can find this channel. Usually, this is the topic name, the routing key, the URL path, etc.
    • So what's this userSignedUp then? This is the channel identifier. It's an identifier that serves to reference the channel from another part of the document or another document.
    • Why not using the address directly? We use JSON Pointer in the $ref construct. Referencing user/signedup using JSON Pointer would be user~1signedup as opposed to user/signedup like many people would think. That's highly unreadable and error-prone, therefore I'm introducing channel identifiers.
    • Channel identifiers also facilitate the task of changing the address in a single place. For instance, if a topic name changes, it has to be changed in only one place and all the other files referencing this channel definition will be automatically updated.
  • Operations object: it's the place where all the operations of this application are defined. channel hints against which channel is this operation performed and action is the type of operation, i.e., we're "sending" or "receiving".
    • What's this sendUserSignedUpEvent? It's the operation identifier. Yes, the old operationId is now mandatory and it's implicit, i.e., the operationId field doesn't exist anymore.
    • Why isn't channel a $ref or JSON Pointer? To keep things simple. If you're referring to a channel here, it must be defined in the channels object. If we allow $ref here, it means it can be dereferenced and therefore the channel ID would be lost. To make things easier and more consistent, this field is simply a string with the name of the channel ID.
    • Why send and receive? No, it's not that I'm hating publish and subscribe already 😅 I'm using send and receive here to avoid confusion for those thinking that AsyncAPI is only meant to describe pub/sub architectures. I think "send" and "receive" are pretty common verbs that shouldn't be linked to any super-very-special meaning.

Organization and reusability at its best

I'm adding two new objects to the components object: servers and channels. Some may be already wondering "why? don't we have servers and channels already in the root of the document?". To understand this decision, I think it's better if I just describe the reusability model I have in mind.

Reusability

Some people expressed their interest in having a "menu" of channels, messages, servers, etc. They're not really interested in a specific application. In other words, they want an organization-wide "library". This is the components object. It's now possible to do something like the following:

asyncapi: 3.0.0

info:
  title: Organization-wide definitions
  version: 3.4.22

components:
  servers:
    ...
  channels:
    ...
  messages:
    ...

This is now a valid AsyncAPI file and it can be referenced from other AsyncAPI files:

asyncapi: 3.0.0

info:
  title: My MQTT client
  version: 3.1.9

servers:
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/servers/mosquitto'

channels:
  userSignedUp:
    $ref: 'common.asyncapi.yaml#/components/channels/userSignedUp'

operations:
  sendUserSignedUpEvent:
    channel: userSignedUp
    action: send

As you can see, I'm not defining any channel or server in this file but instead, I'm pointing to their definitions in the org-wide document (common.asyncapi.yaml). And this leads me to the other part of this section: "organization".

Organization

The example above shows how we explicitly reference the resources (servers and channels) that our application is using. Having them inside components doesn't mean they are making any use of it. And that's key. The file is split into two "sections": application-specific and reusable items. Let's see an example:

asyncapi: 3.0.0

##### Application-specific #####
info: ...
servers: ...
channels: ...
operations: ...
##### Reusable items ######
components:
  servers: ...
  channels: ...
  schemas: ...
  messages: ...
  securitySchemes: ...
  parameters: ...
  correlationIds: ...
  operationTraits: ...
  messageTraits: ...
  serverBindings: ...
  channelBindings: ...
  operationBindings: ...
  messageBindings: ...
###########

So for those of you who were wondering before "why? don't we have servers and channels already in the root of the document?": This is the reason, it's no different than it's right now in v2.x but I thought I'd make it more clear this time. Just because we put something in components doesn't mean the application is making any use of it.

Life is better with examples

Microservices

Say we have a social network, a very basic one. We want to have a website, a backend WebSocket server that sends and receives events for the UI to update in real-time, a message broker, and some other services subscribed to some topics in the broker.

We'll define everything that's common to some or all the applications in a file called common.asyncapi.yaml. Then, each application is going to be defined in its own AsyncAPI file, following the template {app-name}.asyncapi.yaml.

The common.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Organization-wide stuff
  version: 0.1.0

components:
  servers:
    websiteWebSocketServer:
      url: ws://mycompany.com/ws
      protocol: ws
    mosquitto:
      url: mqtt://test.mosquitto.org
      protocol: mqtt

  channels:
    commentLiked:
      address: comment/liked
      message:
        ...
    likeComment:
      address: likeComment
      message:
        ...
    commentLikesCountChanged:
      address: comment/{commentId}/changed
      message:
        ...
    updateCommentLikes:
      address: updateCommentLikes
      message:
        ...
The backend.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Website Backend
  version: 1.0.0

servers:
  websiteWebSocketServer:
    $ref: 'common.asyncapi.yaml#/components/servers/websiteWebSocketServer'
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/servers/mosquitto'

channels:
  commentLiked:
    $ref: 'common.asyncapi.yaml#/components/channels/commentLiked'
  likeComment:
    $ref: 'common.asyncapi.yaml#/components/channels/likeComment'
  commentLikesCountChanged:
    $ref: 'common.asyncapi.yaml#/components/channels/commentLikesCountChanged'
  updateCommentLikes:
    $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes'

operations:
  onCommentLike:
    action: receive
    channel: likeComment
    description: When a comment like is received from the frontend.
  onCommentLikesCountChange:
    action: receive
    channel: commentLikesCountChanged
    description: When an event from the broker arrives telling us to update the comment likes count on the frontend.
  sendCommentLikesUpdate:
    action: send
    channel: updateCommentLikes
    description: Update comment likes count in the frontend.
  sendCommentLiked:
    action: send
    channel: commentLiked
    description: Notify all the services that a comment has been liked.
The frontend.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Website WebSocket Client
  version: 1.0.0

servers:
  websiteWebSocketServer:
    $ref: 'common.asyncapi.yaml#/components/servers/websiteWebSocketServer'

channels:
  likeComment:
    $ref: 'common.asyncapi.yaml#/components/channels/likeComment'
  updateCommentLikes:
    $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes'

operations:
  sendCommentLike:
    action: send
    channel: likeComment
    description: Notify the backend that a comment has been liked.
  onCommentLikesUpdate:
    action: receive
    channel: updateCommentLikes
    description: Update the UI when the comment likes count is updated.
The notifications-service.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Notifications Service
  version: 1.0.0

servers:
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/servers/mosquitto'

channels:
  commentLiked:
    $ref: 'common.asyncapi.yaml#/components/channels/commentLiked'

operations:
  onCommentLiked:
    action: receive
    channel: commentLiked
    description: When a "comment has been liked" message is received, it sends an SMS or push notification to the author.
The comments-service.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.

servers:
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/servers/mosquitto'

channels:
  commentLiked:
    $ref: 'common.asyncapi.yaml#/components/channels/commentLiked'
  commentLikesCountChanged:
    $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes'

operations:
  onCommentLiked:
    action: receive
    channel: commentLiked
    description: Updates the likes count in the database and sends the new count to the broker.
  sendCommentLikesUpdate:
    action: send
    channel: commentLikesCountChanged
    description: Sends the new count to the broker after it has been updated in the database.

Public-facing API

Another common use case for AsyncAPI is to provide a definition of a public-facing API. Examples of this are Slack, Gitter, and Gemini.

This would work differently than it is now. Instead of defining the server as we do with OpenAPI (and AsyncAPI v2), we'd have to define how a client would look like. For instance:

asyncapi: 3.0.0

servers:
  production:
    url: wss://api.gemini.com
    protocol: wss

channels:
  v1MarketDataSymbol:
    address: /v1/marketdata/{symbol}
    parameters:
      ...

operations:
  onMarketSymbolUpdate:
    action: receive
    channel: v1MarketDataSymbol
    description: Receive market updates on a given symbol.

Currently, in v2, the spec lets you describe the server and infer the client from this description (as with OpenAPI). However, this causes a discrepancy with servers, especially in production systems. For instance, my server may be listening on port 5000 because it's behind a proxy. The client should be sending requests/messages to port 80 and they'll be forwarded by the proxy. If we define our port as 80 in the AsyncAPI file and generate a client, everything is ok but if we generate a server, our code will try to listen in port 80, which is not what we want. This means we can't infer clients from server definitions.

The drawback is that we'd have to define 2 files, one for the server and one for the client but the benefit is that we can be 100% accurate with our intents. Tooling can help auto-generating a client definition from a server definition.

Further expansion

Override channel messages with operation-specific messages

The new operations object would help us define additional use cases, like those where we define what message is sent or received to/from a channel at the operation level. Let's see an example:

channels:
  commentLikesCountChanged:
    message:
      oneOf:
        - $ref: 'common.asyncapi.yaml#/components/messages/updateCommentLikes'
        - $ref: 'common.asyncapi.yaml#/components/messages/updateCommentLikesV2'

operations:
  sendCommentLikesUpdate:
    action: send
    channel: commentLikesCountChanged
    description: Sends the new count to the broker after it has been updated in the database.
  sendCommentLikesUpdateV2:
    action: send
    channel: commentLikesCountChanged
    message:
      $ref: 'common.asyncapi.yaml#/components/messages/updateCommentLikesV2'
    description: Does the same as above but uses version 2 of the message.

Adding request/reply support

Adding support for the so-demanded request/reply pattern, would be as easy as adding a new verb and a reply keyword. See example:

channels:
  users:
    message:
      oneOf:
        - $ref: 'common.asyncapi.yaml#/components/messages/createUser'
        - $ref: 'common.asyncapi.yaml#/components/messages/userCreated'

operations:
  createUser:
    action: request
    channel: users
    message:
      $ref: 'common.asyncapi.yaml#/components/messages/createUser'
    description: Creates a user and expects a response in the same channel.
    reply:
      message:
        $ref: 'common.asyncapi.yaml#/components/messages/userCreated'

This is just an example. We should also consider dynamic channel names created at runtime and probably other stuff.

FAQ

Does it serve as a base ground to meet our vision without a future major version?

I think so. Whenever we're ready to support REST, GraphQL, and RPC APIs, this structure should perfectly serve as a base ground.

Does it enable for better reusability of channels and other elements?

Yes. It completely decouples channels from operations and even clarifies the reusability model of the specification.

Does it allow users to define their whole architecture in a single file?

No, but it allows users to have a "common resources" AsyncAPI file where most of the information can reside.

Is it backward-compatible?

Absolutely not.

Does it remove the publish/subscribe confusion without introducing a new confusing term?

Yes. We keep the basic terms as before with just a bit of reorganization.

Is it easy to define a broker-based microservices architecture?

Yes. I provided an example.

Is it easy to define a public asynchronous API?

Yes. I provided an example.

Does it set the base ground to define an RPC (Remote Procedure Call) system over a message broker?

Yes. I provided a basic non-normative example.

Does it set the base ground to define a point-to-point RPC (Remote Procedure Call) API?

Yes. I provided a basic non-normative example.

@magicmatatjahu
Copy link
Member

magicmatatjahu commented Sep 6, 2021

@fmvilas Awesome proposal! I love it ❤️ However, I have some problems with some parts of this proposal, maybe some things are obvious for you and for other people, but some things surprise me and I think their assumptions are wrong.

The main problem - message(s) in the channel

Why do you want to define the message(s) in the channel, not as currently in the operation level? Consider the first example you gave:

asyncapi: 3.0.0

channels:
  userSignedUp:
    address: user/signedup
    message:
      payload:
        type: object
        properties:
          email:
            type: string
            format: email

operations:
  sendUserSignedUpEvent:
    channel: userSignedUp
    action: send
  • How can I guess that by sending a given message the application underneath will process the sendUserSignedUpEvent operation? This problem is more pronounced in the case of Override channel messages with operation-specific messages case.
  • Go ahead. In other examples you defined on the channel the servers (and remotes field) with names of the existing servers. I guess that given channel is only available for particular server/remote. Ok... but what in case when someone want to define multiple operations (with receive kind) for the different servers?
  • What in case when for given channel you wanna define send and receive operation (we have still possibility in the 2.X.X AsyncAPI). If you define the message as oneOf, then how to guess which message is for send and which for receive?

I think that in this case, message(s) should be only defined on the operation, not on the channel. Channels should be then treated only as "interface", which you must "implement" by the operations (also if you need by multiple operations). For me, in the current proposal, the operation is "hardcoded" to the given channel by the message. If I'm not right, please clarify the message field on the channel, because it really doesn't fit for me.

Splitting servers to the two groups

I like idea to clarify that given server is remote and application is connecting to it and another server "runs" the app, but... did you consider having kind in the Server object? Then we don't need the remotes Object. Also as I see, the remotes have this same shape as servers, so kind: remote (or remote: true) is enough for me.

Servers and remotes in the channel

I mentioned about it the first section:

... In other examples you defined on the channel the servers (and remotes field) with names of the existing servers. I guess that given channel is only available for particular server/remote...

but I think that you should clarify this thing in the proposal, because I know that you copied this from existing PR, but people can be surprised 😄

Adding request/reply support

Here I'm just going to think out loud because I'm not an expert in this topic. Maybe instead of an extra action type like request and the replay field we should reuse receive action and add the word operations or then, that would determine what should happen when a given operation is processed and whether it "produces" subsequent operations? Something like, if the payment in the e-shop was successful, send (reply to the broker) an operation that saves record in the database - otherwise create another operation that does rollback etc. This way we could support many more techniques than just request/reply.

Example:

operations:
  checkPayment:
    channel: payment
    action: receive
    then:
      2XX: ... # "reply" operation when given operations success.
      4XX:... # "reply" operation when given operations fails.

In the example above, I use HTTP statuses to describe the state of the operation: success or failure. Probably something better can be invented.

It seems more generic to me. The idea itself is taken from CQRS.

👏🏼 for the proposal. If we go along with this (and I think we will after a dozen or so revisions), then every tool will have to be re-written from scratch 😂

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

Thanks for the quick feedback, Maciej! 🙌

Why do you want to define the message(s) in the channel, not as currently in the operation level?

At least 2 reasons come to my mind:

  • Some people want to use AsyncAPI as a menu of channels and the messages one can expect there. Mostly for discovery. It makes sense then to have messages associated with channels because there will not be operations for them.
  • I think it's a good practice to establish what are the expected messages in a specific channel, regardless of who's sending them. This way, you can make sure the list of messages is well-defined and controlled. If the messages are defined in operations only, it becomes chaotic and you'd need to explore all the AsyncAPI files to know which messages you can expect in a given channel. By defining them in the channel, it can be defined in a single place.

Channels should be then treated only as "interface", which you must "implement" by the operations (also if you need by multiple operations).

Precisely. Interfaces let you define the "shape" of the implementation and that's what we're doing here.

For me, in the current proposal, the operation is "hardcoded" to the given channel by the message. If I'm not right, please clarify the message field on the channel, because it really doesn't fit for me.

How is that hardcoded? We're just standardizing what messages can go through a channel and I think this is actually one of the biggest values. It's exactly how it is today. In v2, the message is tied to an operation and a channel at the same time in the same place.

How can I guess that by sending a given message the application underneath will process the sendUserSignedUpEvent operation? This problem is more pronounced in the case of Override channel messages with operation-specific messages case.

Mind clarifying this? I'm not sure I understand your question/concern.

Go ahead. In other examples you defined on the channel the servers (and remotes field) with names of the existing servers. I guess that given channel is only available for particular server/remote. Ok... but what in case when someone want to define multiple operations (with receive kind) for the different servers?

Sorry, that was just an experiment I was doing and forgot to delete it before publishing. Thanks for the heads up.

What in case when for given channel you wanna define send and receive operation (we have still possibility in the 2.X.X AsyncAPI). If you define the message as oneOf, then how to guess which message is for send and which for receive?

You can define as many operations as you want in a given channel :)

operations:
  mySendOperation:
    channel: myChannel
    ...
  myReceiveOperation:
    channel: myChannel
    ...
  myOtherReceiveOperation:
    channel: myChannel
    ...

I like idea to clarify that given server is remote and application is connecting to it and another server "runs" the app, but... did you consider having kind in the Server object?

I did, and instantly discarded it 😄 When you're referencing a server or a remote, you can clearly know by the URL if it's either a server o a remote. E.g.:

$ref: 'common.asyncapi.yaml#/components/servers/production' # I know it's a server just by looking at the URL.
$ref: 'common.asyncapi.yaml#/components/remote/production' # I know it's a remote just by looking at the URL.

In general, just because two things have the same shape, it doesn't mean they have to be merged. Structurally they look the same but semantically they're completely different and subject to evolve in different directions in the future.

Maybe instead of an extra action type like request and the replay field we should reuse receive action and add the word operations or then, that would determine what should happen when a given operation is processed and whether it "produces" subsequent operations?

Not a concern of this proposal (my example was just to showcase that something can be done) but I'll bear with you. It seems what you're trying to define there is workflow language. And that's a totally different thing altogether 😄


Thanks a lot, Maciej! I hope this clarifies things a bit more :)

@iancooper
Copy link

iancooper commented Sep 7, 2021

I think this is a very positive proposal and helps with a lot of the issues around re-use of channels in typical microservice documentation etc.

A quick initial question. My understanding is server is something we have and remote is something we use. With that assumption, should there be an explicit connection between server/remote and channel? If i a channel becomes a reusable asset that I can define in a central file, where it is shared by producer and consumer, where is that channel hosted? Do I want to define the remote that hosts my channel in every file, even if I don't re-declare the channel there to somehow indicate they are hosted there?

On Identifiers, I think there may be a generally useful concept here. Having been working with $ref whilst I try and put together an SNS/SQS binding, it struggles where you want to imply resource uses resource b - defined elsewhere - instead of pull in definition of resource b.

@magicmatatjahu
Copy link
Member

@fmvilas Thanks for explanation. Some things are clear but message at the channel level doesn't fit me 😄

In v2, the message is tied to an operation and a channel at the same time in the same place.

Yes, I agree but currently you define given message(s) for particular action/operation (publish/subscribe), even if you have defined explicit operation in the channel level. I can understand the reason for having messages in the channel level, but that's the biggest problem to understand: whether a given message(s) by default is/are associated with send or receive operation?

Also you wrote:

You can define as many operations as you want in a given channel :)

Yep, but you may end up with a message defined at the channel level that fits only one operation, and for the rest you have to define the message at the operation level. So what it value for the message in the channel level? This can only cause problems in understanding what message an operation takes.

If an operation will have the ability to define messages which will override those on the channel level then I can accept this, but I opt for having messages only on the operation level. Can you give me an episode of ThinkingOutLoud or a discussion in which a channel-level message was proposed? Thanks!

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

A quick initial question. My understanding is server is something we have and remote is something we use. With that assumption, should there be an explicit connection between server/remote and channel? If i a channel becomes a reusable asset that I can define in a central file, where it is shared by producer and consumer, where is that channel hosted? Do I want to define the remote that hosts my channel in every file, even if I don't re-declare the channel there to somehow indicate they are hosted there?

There's an ongoing proposal for that. I didn't want to include it here on purpose to avoid bloating this issue so much: #531

Yes, I agree but currently you define given message(s) for particular action/operation (publish/subscribe), even if you have defined explicit operation in the channel level. I can understand the reason for having messages in the channel level, but that's the biggest problem to understand: whether a given message(s) by default is/are associated with send or receive operation?

That's the whole point. Not everyone is interested in operations. Many just want a menu of channels and messages and they don't care who sends or receives them.

So what it value for the message in the channel level? This can only cause problems in understanding what message an operation takes.

The value is the one I described above. Those who want a menu of channels and their associated messages. Many people don't want to describe applications but their sources of data (topics aka channels).

If an operation will have the ability to define messages which will override those on the channel level then I can accept this

Yes. An operation has the ability to override a channel's message definition. Now the question is, should we allow to completely override it? Or should the operation's message be a subset of the channel's message? 🤔 The first gives you more freedom but potentially overcomplicates things. The second is more restrictive but gives you consistency.

Can you give me an episode of ThinkingOutLoud or a discussion in which a channel-level message was proposed? Thanks!

It wasn't proposed like this explicitly but here's a conversation with @dalelane about it: https://youtu.be/Qsu_yC-5YYM?t=535. I recommend you watch the whole episode.

@jonaslagoni
Copy link
Member

jonaslagoni commented Sep 7, 2021

Is there a reason we have a channels field in the root AsyncAPI object, would it not be enough to define it inside the operation object and components? 🤔

The reason behind this is that I hate you have to manually match channel id's, if it can be avoided.

Gonna go with your examples to show the changes, remember in this case $ref can be switched out completely with the channel object itself if you don't want to use that feature.

This will help us in tooling, as:

  1. You don't have to verify that operations have valid channels in runtime.
  2. From a parser perspective, you don't have to manually match operations with channels.

And it still enables all the same things, but without complexity.

The common.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Organization-wide stuff
  version: 0.1.0

components:
  servers:
    websiteWebSocketServer:
      url: ws://mycompany.com/ws
      protocol: ws

  remotes:
    mosquitto:
      url: mqtt://test.mosquitto.org
      protocol: mqtt

  channels:
    commentLiked:
      address: comment/liked
      message:
        ...
    likeComment:
      address: likeComment
      message:
        ...
    commentLikesCountChanged:
      address: comment/{commentId}/changed
      message:
        ...
    updateCommentLikes:
      address: updateCommentLikes
      message:
        ...
The backend.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Website Backend
  version: 1.0.0

servers:
  websiteWebSocketServer:
    $ref: 'common.asyncapi.yaml#/components/servers/websiteWebSocketServer'

remotes:
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/servers/mosquitto'

operations:
  onCommentLike:
    action: receive
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/likeComment'
    description: When a comment like is received from the frontend.
  onCommentLikesCountChange:
    action: receive
    channel:
    	$ref: 'common.asyncapi.yaml#/components/channels/commentLikesCountChanged'
    description: When an event from the broker arrives telling us to update the comment likes count on the frontend.
  sendCommentLikesUpdate:
    action: send
    channel: 
   	 $ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes'
    description: Update comment likes count in the frontend.
  sendCommentLiked:
    action: send
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/commentLiked'
    description: Notify all the services that a comment has been liked.
The frontend.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Website WebSocket Client
  version: 1.0.0

remotes:
  websiteWebSocketServer:
    $ref: 'common.asyncapi.yaml#/components/servers/websiteWebSocketServer'

operations:
  sendCommentLike:
    action: send
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/likeComment'
    description: Notify the backend that a comment has been liked.
  onCommentLikesUpdate:
    action: receive
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes'
    description: Update the UI when the comment likes count is updated.
The notifications-service.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Notifications Service
  version: 1.0.0

remotes:
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/remotes/mosquitto'

operations:
  onCommentLiked:
    action: receive
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/commentLiked'
    description: When a "comment has been liked" message is received, it sends an SMS or push notification to the author.
The comments-service.asyncapi.yaml file
asyncapi: 3.0.0

info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.

remotes:
  mosquitto:
    $ref: 'common.asyncapi.yaml#/components/remotes/mosquitto'

operations:
  onCommentLiked:
    action: receive
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/commentLiked'
    description: Updates the likes count in the database and sends the new count to the broker.
  sendCommentLikesUpdate:
    action: send
    channel: 
    	$ref: 'common.asyncapi.yaml#/components/channels/updateCommentLikes'
    description: Sends the new count to the broker after it has been updated in the database.

Do you see any reason we would not do this? 🤔

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

Is there a reason we have a channels field in the root AsyncAPI object, would it not be enough to define it inside the operation object? 🤔

Because many people don't want to use AsyncAPI to define applications but a menu of channels, messages, servers, etc. Systems like event gateways are not interested in operations at all but in which channels and messages are available.

@jonaslagoni
Copy link
Member

jonaslagoni commented Sep 7, 2021

Because many people don't want to use AsyncAPI to define applications but a menu of channels, messages, servers, etc. Systems like event gateways are not interested in operations at all but in which channels and messages are available.

@fmvilas forgot to add components as well. Components still give you the menu? Of course, the channels now become secondary, instead of one of the primary attributes of the AsyncAPI file but it still enables the menu.

We can of course just keep the channels field, to enable the primary behavior. But I see no reason why operations must manually define the matching of channel id's. I just really want to remove this (manual) runtime matching of channels if possible 😬

@magicmatatjahu
Copy link
Member

magicmatatjahu commented Sep 7, 2021

@jonaslagoni You can read my comment about matching channels to the servers and why operating on the id (names) of the server/channel is better option rather than refs (and plain Server/Channel Object) -> #531 (comment)

@fmvilas Thanks, I'm starting to see that. With the proposal, we have an options:

  • treat the AsyncAPI document as information about the channels and "connected" messages themselves (for gateways)
  • treat the AsyncAPI document as describing the operations that are performed within the application, which can also be connected to a gateway.

Additionally, if messages were not defined at the channel level, but only at the operation level, we would have to have a tool that would retrieve these messages from operations - most likely we will have to have such a tool due to the fact that operations will be able to override messages from the channel.

Yes. An operation has the ability to override a channel's message definition. Now the question is, should we allow to completely override it? Or should the operation's message be a subset of the channel's message? 🤔 The first gives you more freedom but potentially overcomplicates things. The second is more restrictive but gives you consistency.

The second option is better and more consistency. I don't know how to handle situation when you have defined on the channel level the 5 messages (by oneOf) and in operation you wanna say that only second operation is handled, or all other without this second...

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

@jonaslagoni you're right. Actually, the channels field in the root object is not for "menu" purposes but to define which channels this application is making use of, so forget my previous comment. I see not blocking reason for what you're saying. One reason I can think of is that, to know which channels are being used in an application, we'd have to scan all the operations first. Also, by having this matching we make it mandatory to reuse channel definitions, making it more difficult for people to duplicate stuff. An idea that comes to my mind is that we can disallow dereferencing there and instead you can only use $ref but that complicates things on the tooling side. And another reason is that we'd lose the channel id after dereferencing. Not sure if it's a problem though.

In general, I'd not worry about tooling so much. Let's put the focus on the user experience. If it's better for the user, cool let's do it, if it's not, let's change it. Our decisions should be driven by users and not tooling difficulty/complexity. That said, there may be some good for UX in what you're proposing, especially that it would work with existing JSON Schema tools in editors.

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

Additionally, if messages were not defined at the channel level, but only at the operation level, we would have to have a tool that would retrieve these messages from operations - most likely we will have to have such a tool due to the fact that operations will be able to override messages from the channel.

@magicmatatjahu Yes, we'd need such a tool but some people might not need it. E.g., those who don't use the operations keyword.

The second option is better and more consistency. I don't know how to handle situation when you have defined on the channel level the 5 messages (by oneOf) and in operation you wanna say that only second operation is handled, or all other without this second...

Maybe we should make the channel's message work with message ids? This way we enforce messages having ids, which is another feature request: #458. It could look something like this:

channels:
  myChannel:
    address: my/channel
    messages:
      - myMessage
      - myMessage2
      - myMessage3

operations:
  sendMyMessage2:
    channel: myChannel
    messages:
      - myMessage2

messages:
  myMessage:
    payload:
      ...
  myMessage2:
    payload:
      ...
  myMessage3:
    payload:
      ...

This would make it possible to check if myMessage2 is part of the channel's messages definition.

@magicmatatjahu
Copy link
Member

magicmatatjahu commented Sep 7, 2021

@fmvilas Yes, it's a solution. However I see in this solution and with referencing the channels by $ref (not by id/names) some problems (@jonaslagoni).

If I have a situation when a given broker has in "menu" some channel and then I connect this channel in operation (in some app connected to broker), how do I know if a channel is unique in the system? To describe this in more detail, I will use an example. I have one broker and one application that is connected to the broker:

The broker.yaml file
asyncapi: 3.0.0

info:
  title: Broker
  version: 1.0.0

servers:
  brokerServer:
    ...

channels: 
  someChannel1:
    ...
  someChannel2:
    ...
The app1.yaml file
asyncapi: 3.0.0

info:
  title: App1
  version: 1.0.0

servers:
  someServer:
    ...

remotes:
  brokerRemote:
    $ref: 'broker.yaml#/servers/brokerServer'

operations:
  someOperation1:
    channel: 
      $ref: 'broker.yaml#/channels/someChannel1'
  someOperation2:
    channel: 
      $ref: 'broker.yaml#/channels/someChannel2'

How can I tell that a particular channel is unique in the system and that is what it describes operations to? Even if we use channel identifiers, we still have a problem, because the identifier itself in application 1 may be different than the identifiers in the broker, where it is defined:

The broker.yaml file
asyncapi: 3.0.0

info:
  title: Broker
  version: 1.0.0

servers:
  brokerServer:
    ...

channels: 
  someChannel1:
    ...
  someChannel2:
    ...
The app1.yaml file
asyncapi: 3.0.0

info:
  title: App1
  version: 1.0.0

servers:
  someServer:
    ...

remotes:
  brokerRemote:
    $ref: 'broker.yaml#/servers/brokerServer'

channels: 
  anotherIdForChannel1:
    $ref: 'broker.yaml#/channels/someChannel1'
  anotherIdForChannel2:
    $ref: 'broker.yaml#/channels/someChannel2'

operations:
  someOperation1:
    channel: anotherIdForChannel1
  someOperation2:
    channel: anotherIdForChannel2

It seems to me that the operation and the channel in its object should have something like operationID and channelID and these values should be unique in the whole system - then we have an easy way to refer to channels.

The broker.yaml file
asyncapi: 3.0.0

info:
  title: Broker
  version: 1.0.0

servers:
  brokerServer:
    ...

channels: 
  someChannel1:
    channelID: broker-someChannel1
    ...
  someChannel2:
    channelID: broker-someChannel2
    ...
The app1.yaml file
asyncapi: 3.0.0

info:
  title: App1
  version: 1.0.0

servers:
  someServer:
    ...

remotes:
  brokerRemote:
    $ref: 'broker.yaml#/servers/brokerServer'

operations:
  someOperation1:
    channel: 
      $ref: 'broker.yaml#/channels/someChannel1'
  someOperation2:
    channel: 
      $ref: 'broker.yaml#/channels/someChannel2'

The same problem occurs with the messages, how do I know if a message is unique in the system?

I deliberately did not use the channel's address because it can change (as was mentioned by Fran) and the channelID itself should not.

Maybe I'm talking crap and it's not necessary 😆

EDIT: Ok 😄 on second thought, address can be treated as a channelID and references make sense here.

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

@magicmatatjahu I think you're getting a few things wrong here. Let me recap:

# broker.yaml

asyncapi: 3.0.0

info:
  title: Broker
  version: 1.0.0

servers:
  brokerServer:
    ...

channels: 
  someChannel1:
    ...
  someChannel2:
    ...

This definition is not semantically valid. This is not a menu, it's an application definition. Menus can only use components. That's the reason we now have components.channels. This is how you'd define a menu:

# broker.yaml

asyncapi: 3.0.0

info:
  title: Broker
  version: 1.0.0

components:
  servers:
    brokerServer:
      ...
  channels: 
    someChannel1:
      ...
    someChannel2:
      ...

Also, your broker is not a server of your applications but a remote so it should be like this:

# broker.yaml

asyncapi: 3.0.0

info:
  title: Broker
  version: 1.0.0

components:
  remotes:
    broker:
      ...
  channels: 
    someChannel1:
      ...
    someChannel2:
      ...

Now we got a broker menu. Let's define app1.yaml:

asyncapi: 3.0.0

info:
  title: App1
  version: 1.0.0

remotes:
  broker: # Notice how I'm using a remote and it's referencing a remote too.
    $ref: 'broker.yaml#/components/remotes/broker'

channels:
  channel1: # Notice this Id is different than the one in broker.yaml. Ids are local to the AsyncAPI file.
    $ref: 'broker.yaml#/components/channels/someChannel1'
  channel2:
    $ref: 'broker.yaml#/components/channels/someChannel2'

operations:
  someOperation1:
    channel: channel1
  someOperation2:
    channel: channel2

This is how you know that App1 is using channel1 and channel2, because they're being used in the channels object. You don't need unique ids in the whole system because their ids are local to the file and that's ok. channel1 and channel2 are the same as someChannel1 and someChannel2 respectively. I just chose to use a different id to illustrate that ids are local. Does it make sense?

@magicmatatjahu
Copy link
Member

@fmvilas Thanks for explanation, it makes sense 😅 Even if you are using the local id of the channels, you should have information (especially in tooling like our cupid) that the given channel is unique and for this you can use the channel's address and then we have connections:

channel1 -> broker.yaml#/components/channels/someChannel1 -> channel's address

I edited my previous comment, maybe you read it too late.

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

Yes, the address can be the global id.

@jessemenning
Copy link

jessemenning commented Sep 7, 2021

I think this is a very positive proposal and helps with a lot of the issues around re-use of channels in typical microservice documentation etc.

A quick initial question. My understanding is server is something we have and remote is something we use.

I agree with @iancooper on both points, it's a positive proposal and I have concerns about the definition of server and remote. Because depending on the perspective, the same underlying object is a both a server and a remote.

For example: a websocket client connecting to a node.js application that spawns an internal websocket server. The node.js application considers the websocket server an server. The client considers the same websocket server to be an remote.

Concerns

  • The websocket server needs to be defined twice in AsyncAPI as both a server and an remote even though its the same underlying entity
  • As we migrate through environments, the definitions of remote and server need to be updated in lockstep.
  • We need to explain to new developers what the difference is between a remote and a server, which is not entirely obvious, along the lines of the current publish/subscribe perspective issue.
  • We've introduced a top level element that is not applicable to a large subset of async implementations.

This seems contrary our aims of addressing the perspective issue, the re-use issue, and moving implementation specific information out of core objects.

Alternative
It also seems that given that this is largely a protocol-specific issue, the answer lies in protocol-specific bindings. @fmvilas points out there are code generation implications here--websocket implementations don't know whether to generate an internal websocket server or a client-connection to an external broker. As an alternative, I would suggest that there be separate websocket bindings (call it websocket-client and websocket-server) that hint to the code generator how to implement the particular scenario. This would seem to address the concerns raised in the most minimal way possible.

This extends the (key in my mind) effort to make the core objects of AsyncAPI as close to purely logical as possible. Servers house node-level connection information and can be grouped together into Environments. These concepts are applicable to all async interactions.

@jessemenning
Copy link

jessemenning commented Sep 7, 2021

Is it backward-compatible?

Absolutely not.

Concerns
I think this deserves a lot of thought and discussion. As @GeraldLoeffler states, as a 2.1 specification, there is an expectation from both our community and general public that the spec is relatively stable. And many find the current spec to be perfectly suitable for their use cases. My feeling would be incompatible changes should be the last resort.

Internally I know a lot of large (=> less agile) enterprises that have coded to the 2.1 spec. To have that code be incompatible with 3.0 would likely decrease AsyncAPI's momentum.

And yes, we can maintain separate 2.x and 3.x branches, but we all know the challenges of maintaining two versions of code, even in the medium term. This is especially true when there's a young spec with an imbalance between tooling needs and developers to implement them.

Alternative
Given that v2.1 makes very specific assumptions about the world (e.g. it describes the perspective of the application, the client is the mirror image of the application), in the absence of v3.0 terminology, the new version "falls back" to the v2 interpretation of objects. I feel like this is similar in spirit to @lornajane OpenAPI's introduction of websockets--if websockets aren't included, the typical behavior of OpenAPI is observed.

@fmvilas
Copy link
Member Author

fmvilas commented Sep 7, 2021

Because depending on the perspective, the same underlying object is a both a server and a remote.

Alright. You gave me an idea and I changed the proposal. Now components only have servers and not remotes, since a remote is a server in the end. We still keep the root remotes object because —from the point of view of the application— it still makes sense. And let's not forget that an AsyncAPI file defines an application unless it only has a components object.

The websocket server needs to be defined twice in AsyncAPI as both a server and an remote even though its the same underlying entity

That's not true. A remote can reference a server object. It was also like this before I made the change described above. Actually, my "microservices" example shows this possibility when the frontend is using a remote that's referencing a WebSocket server definition.

As we migrate through environments, the definitions of remote and server need to be updated in lockstep.

That should be solved by my latest change 👍

We need to explain to new developers what the difference is between a remote and a server, which is not entirely obvious, along the lines of the current publish/subscribe perspective issue.

We needed this change anyway, even in version 2.

We've introduced a top level element that is not applicable to a large subset of async implementations.

Which one? 🤔

It also seems that given that this is largely a protocol-specific issue, the answer lies in protocol-specific bindings.

I'm using WebSockets as examples but it has nothing to do with WebSockets only. HTTP servers, HTTP2 SSE, GraphQL subscriptions, and more will have the same problem. It has nothing to do with the protocol but with the architecture design. Some interactions are client/server and some are broker-based. When it's client/server, we need to make it clear if the server is "something we're exposing" or "something we're connecting to". I don't think it's practical nor elegant to have bindings like websockets-cilent, websockets-server, http-server, http-client, graphql-client, graphql-server, and so on. That actually sounds like a huge hack/patch to me.

And yes, we can maintain separate 2.x and 3.x branches, but we all know the challenges of maintaining two versions of code, even in the medium term.

I don't think we'd even need to maintain two branches of the spec but instead, we can and have to maintain tooling compatibility with the two major versions. Until v2.x gets deprecated and we don't support it anymore but that can be in 1 or 2 years (or whatever we decide). If someone wants to stay in v2, cool! we'll keep supporting it. If you want to get the most out of the spec and tooling, migrate to v3. I think it's a fair and common thing.

As @GeraldLoeffler states, as a 2.1 specification, there is an expectation from both our community and general public that the spec is relatively stable.

This is especially true when there's a young spec with an imbalance between tooling needs and developers to implement them.

Are we young or stable? 😄 I think we're still young and it's true we're gaining momentum but that shouldn't stop us from evolving without ending up with a Frankenstein specification with tons of band-aids just not to break backward compatibility. The users would love to have something that's clear and beautiful to use. IMHO, it's a problem of tooling vendors (including us) to migrate the tools to give the best experience to the users.

Regarding imbalance between tooling needs and developers to implement them, we're working on this. At Postman we're hiring a bunch of people to work exclusively on AsyncAPI. Maybe other companies should follow. Also, the community is growing super fast so that imbalance we'll soon be equilibrated.

@dalelane
Copy link
Collaborator

dalelane commented Sep 8, 2021

I'm still digesting all of this, so this isn't a very considered response from me.

One initial thought jumped out at me though:
How do we represent distributed multi-broker systems like Kafka?

e.g. If I have a Kafka cluster with three brokers I might start with:

remotes:
  broker1:
    url: broker1.myhost.com
    protocol: kafka
  broker2:
    url: broker2.myhost.com
    protocol: kafka
  broker3:
    url: broker3.myhost.com
    protocol: kafka

That gives me three brokers to have to refer to elsewhere/in other specs, which feels clunky.

I know we've discussed this idea of groups of servers before, so I think it would be good to resolve this issue as part of a jump to 3.0

It doesn't necessarily have to be over-engineered. Kafka uses bootstrap addresses which are made of combining the broker URLs into a list, so we could do something like that.

remotes:
  mycluster:
    url: broker1.myhost.com,broker2.myhost.com,broker3.myhost.com
    protocol: kafka

or

remotes:
  mycluster:
    urls: 
      - broker1.myhost.com
      - broker2.myhost.com
      - broker3.myhost.com
    protocol: kafka

It sort of breaks the idea of this really being a "url", but it does simplify things:

  • gives us one object to refer to and re-use
  • means we don't have to duplicate things like protocol and security scheme, which would always have to be set the same for all brokers in a group anyway
  • makes code generation more straightforward, as they would only have to combine the broker addresses anyway

@jessemenning
Copy link

jessemenning commented Sep 8, 2021

Alright. You gave me an idea and I changed the proposal. Now components only have servers and not remotes, since a remote is a server in the end. We still keep the root remotes object because —from the point of view of the application— it still makes sense. And let's not forget that an AsyncAPI file defines an application unless it only has a components object.

That improves re-use, but harms the user experience. I can envision explaining the difference between a server and a remote to a new developer, and then having to then explain why remote then refers to a server component. It's very reminiscent of the perspective issue.

As we migrate through environments, the definitions of remote and server need to be updated in lockstep.

That should be solved by my latest change 👍

Agreed, but I don't think in an optimal way.

We need to explain to new developers what the difference is between a remote and a server, which is not entirely obvious, along the lines of the current publish/subscribe perspective issue.

We needed this change anyway, even in version 2.

I agree that AsyncAPI needs a way to indicate client or server code implementation. But that does not require the introduction of server and remote verbiage.

We've introduced a top level element that is not applicable to a large subset of async implementations.

Which one? 🤔

More accurate wording would be “redefines”. This proposal redefines server from “high-level connection information” which is widely applicable to all async use cases to “connection information, but only for internally spawned servers, but only when the application acts as server for that particular function”. The redefined object is applicable to a greatly reduced subset of async use cases.

I don't think it's practical nor elegant to have bindings like websockets-cilent, websockets-server, http-server, http-client, graphql-client, graphql-server, and so on. That actually sounds like a huge hack/patch to me.

I would say its practical and common to separate server and client implementations. For example, many jars don't contain both the client and server for a particular technology. And keeping separation of between logical objects and implementation seems like an elegant, non-hacky concept that allows the spec to be vendor neutral and extensible, and allows end users the ability to switch implementations without altering the logical structure of their architecture. But if a proliferation of bindings is a concern, perhaps a third option would be to add a flag to applicable bindings indicating whether the binding is acting as a client or a server.

And yes, we can maintain separate 2.x and 3.x branches, but we all know the challenges of maintaining two versions of code, even in the medium term.

I don't think we'd even need to maintain two branches of the spec but instead, we can and have to maintain tooling compatibility with the two major versions. Until v2.x gets deprecated and we don't support it anymore but that can be in 1 or 2 years (or whatever we decide). If someone wants to stay in v2, cool! we'll keep supporting it. If you want to get the most out of the spec and tooling, migrate to v3. I think it's a fair and common thing.

I hope that v2 continues to receive support, but that my experience is that shiny new things get all of the attention to the detriment of those who committed to earlier versions of the spec.

Are we young or stable? 😄

"v2.1" projects stability. Radical changes from version to version projects "young". The dissonance between the two needs to be considered.

@fmvilas
Copy link
Member Author

fmvilas commented Sep 8, 2021

How do we represent distributed multi-broker systems like Kafka?

@dalelane I think your proposal #465 makes sense and should be included in v3. Actually, everything that's a breaking change should land in v3 and we should be aware of all of them before we release v3, just so we don't release v4 some months later 😄 Let's keep this issue focused on publish/subscribe and its associated problems.

That improves re-use, but harms the user experience. I can envision explaining the difference between a server and a remote to a new developer, and then having to then explain why remote then refers to a server component. It's very reminiscent of the perspective issue.

I understand your concern but I think this remote/server difference is way easier to understand than the problem with publish/subscribe. We could argue the same about channels. We have to explain it because some people call them "topics", some "event name", some "routing keys", etc. We also had to explain it and it's ok so far. As long as what we're explaining is easy to grasp and reason about, I don't see a problem.

I agree that AsyncAPI needs a way to indicate client or server code implementation. But that does not require the introduction of server and remote verbiage.

I'm sorry but I think introducing verbiage like [protocol]-client and [protocol]-server in bindings is actually worse because you still have these concepts but replicated among a bunch of protocols. @magicmatatjahu proposed having a kind attribute that differentiates both and I think it's starting to make more sense now that components/remotes is not a thing anymore. Something like:

servers:
  wsBackendServer:
    url: ...
    kind: local # Defaults to "remote" if not specified.
  mosquitto:
    url: ...

My only argument against it was that it was easier to read something like $ref: 'common.asyncapi.yaml#/components/remotes/mosquitto' and automatically know it's a remote because it's in the URL. But since components/remotes is not a thing anymore this argument doesn't apply anymore.

The redefined object is applicable to a greatly reduced subset of async use cases.

You may be right but this change is meant to serve as the base ground for the future vision, which does not only focus on async stuff.

@ekozynin
Copy link

ekozynin commented Sep 10, 2021

It should be possible to define multiple remotes for the same environments. For example, Kafka cluster will consist of number of remote urls:

  • broker(s)
  • schema registry
  • connector instance(s)
  • KSql instance(s)

The above set of remotes (that logically forms a Kafka cluster) will be repeated for different environments (dev, prod).

Also we should be able to specify some common remote systems (for example Secret manager) per environment.
'servers' can de defined under the environment as well.

Do you think a concept of 'environment' should be introduced on the root level?

environment:
  localDev:
    servers:
        # servers here
    remotes:
      broker:
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

  staging:
    servers:
        # servers here
    remotes:
      broker:
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

@fmvilas
Copy link
Member Author

fmvilas commented Sep 10, 2021

@ekozynin I think this is an interesting proposal that deserves its own separate discussion. Would you mind opening a new issue so we keep this one focused on solving the publish/subscribe confusion? 🙏

@joerg-walter-de
Copy link

joerg-walter-de commented Sep 10, 2021

I am currently developing a websocket communcation system by using AsyncAPI specs. This seems to be the right discussion regarding some issues I see.

Remotes.

Remotes: A remote is a remote server our application will connect to

There should be no our. We design an API. When we implement it we may implement for example a websocket server or websocket clients or both. Therefore remote doesn't make much sense, because it takes the perspective of a client which may be not ours (or even is never ours).

Operations and messages

When I design an OpenAPI spec, I have combinations of paths and methods that are associated with operationIds, e.g. createEntry, that refers to operations createEntry(). The resource is an Entry. This name createEntry makes complete sense on the server side and on the client side. I then especially generate code for the client that will contain a createEntry().

An AsyncAPI spec should allow the automatic generation of server and client code.

A resource in OpenAPI world is losely related to message in AsyncAPI world. E.g. in a websocket connection a client may subscribe and unsubscribe to new entries and will then receive datagrams for new entries. The client therefore will send a subscribe message, an unsubscribe message.

Therefore operations createSubscription, deleteSubscription available to clients in generated client code would be appropriate in OpenAPI world and also in AsyncAPI world. These names would be at least OK in the producer/server domain. A handler function createSubscription() in the server, that handles subscription requests coming from future consumers would be OK.

That said it seems that an operationId createSubscription that is associated with the message for a subscription would be acceptable. Maybe two operationsId s createSubscription and onCreateSubscription that are both associated with that message would be even better.

Channels

What is a channel for example in a websocket system? Currently it is one connection.

I have defined a channel for entries. This is one websocket connection. In this channel a subscription message is send by the consumer and the entry message is send by the producer. But the unsubscribe message is send over the same channel, too. I don't want to open multiple connections for that.

Therefore I want to associate multiple messages with on channel. With the natural association of one (or two) operations per message, it seems that it would be good to have multiple messages associated with one channel (and not via oneOf) and (at least) one operation associated with every single message.

In your draft I don't see that option:

asyncapi: 3.0.0

channels:
  userSignedUp:
    address: user/signedup
    message:
      payload:
        type: object
        properties:
          email:
            type: string
            format: email

operations:
  sendUserSignedUpEvent:
    channel: userSignedUp
    action: send

A channel has one address (i.e. URL, i.e connection with websockets) and still has one message associated. An operation is associated with the channel and not with the message.

Proposal

I propose to associate multiple messages (array, not one of) with one channel and associate one operation with one message in one channel.

On the question of one or two operations per channel/message combination I am torn. It works fine with one for me in OpenApi world, though.

@jessemenning
Copy link

@magicmatatjahu proposed having a kind attribute that differentiates both and I think it's starting to make more sense now that components/remotes is not a thing anymore.

@fmvilas, that seems reasonable to me. Thanks, @magicmatatjahu !

@fmvilas fmvilas self-assigned this Sep 14, 2021
@jfallows
Copy link

jfallows commented Sep 1, 2022

Once the nature of the interactions between systems is captured, we can try to create a model to describe those interactions:

  • The interface should describe the list of operations (commands). For each operation, the interface should describe:

    • The input parameters (if any)

    • The possible results (if any, no matter if sync (request / response for REST APIs) or async (request / reply pattern)). For each possible result:

      • The output parameters (if any)
    • The generated events that must be emitted (if any)

    • The callbacks (as defined in OAS) that must be executed (if any)

The following spec snipped illustrates what such a spec could look like (mix of REST API and Async API):

paths: #OAS 
  /orders:
    post:
      requestBody:
      responses:
      callbacks:
      events: #extending OAS spec
        orderCreated:
          channel:
          messages:

operations: #Async API
  approveOrder:
    request:
      channel:
      message:
    replies: #support for request / reply pattern. If not present, fire and forget.
      success:
        channel:
        message:
      failure:
        channel:
        message:
    callbacks:
    events:
      orderApproved:
        channel:
        messages:
      orderApprovalFailed:
        channel:
        messages:

Does it make sense? What are your thoughts about it?

@olamiral this approach makes a lot of sense.

This describes the surface area of the service interface without confusing the concepts of client, server, sender, receiver, publisher, subscriber, and also shows how AsyncAPI might incorporate a unified approach for all REST and Event-Driven APIs.

What do people think of this aspect of this proposal that is directly relevant to publish/subscribe confusion, independent of the separate request-reply conversation happening in #558 ?

@jfallows
Copy link

jfallows commented Sep 1, 2022

Currently, in v2, the spec lets you describe the server and infer the client from this description (as with OpenAPI). However, this causes a discrepancy with servers, especially in production systems. For instance, my server may be listening on port 5000 because it's behind a proxy. The client should be sending requests/messages to port 80 and they'll be forwarded by the proxy. If we define our port as 80 in the AsyncAPI file and generate a client, everything is ok but if we generate a server, our code will try to listen in port 80, which is not what we want. This means we can't infer clients from server definitions.

This perceived problem might be conflating API definition with deployment concerns, leading to API definition divergence and logical duplication.

There's a lot to be said for having a single source of truth for the surface area of a service interface, and once we go down the path of having multiple definitions from different perspectives such as backend and frontend, that can get lost.

operations:
  #backend
  onCommentLike:
    action: receive
    channel: likeComment
    description: When a comment like is received from the frontend.
operations:
  #frontend
  sendCommentLike:
    action: send
    channel: likeComment
    description: Notify the backend that a comment has been liked.

What then does it mean to evolve this service API?
Do we need to make companion updates to each perspective and keep them in sync?

REST services are naturally relocatable at different origin servers, such as https://example.com/ vs http://localhost:5000/ which makes it easy to do local development or container-based deployment for a service that might be advertised at the http://example.com/ address. REST services are also naturally relocatable at a different root path, such as http://localhost:5000/example/, due to the hierarchical nature of paths in URLs.

Perhaps focusing on where AsyncAPI is already good at providing a relocatable service API definition and seeing where it can be improved would help retain the simplicity of a single source of truth, support client code generation and also support server code generation, potentially parameterized at the point of generation or at runtime for relocated deployments?

The ideas proposed by @olamiral above would seem to be helpful towards that and perhaps worthy of further discussion.

@fmvilas
Copy link
Member Author

fmvilas commented Sep 2, 2022

@jfallows thanks a lot for the feedback. It seems all your comments are really focused on HTTP and REST APIs but these are not the target of AsyncAPI (at least not yet). We're trying to get WebSocket (client-server) and broker-based architectures (Kafka, MQTT, AMQP, etc.) right first.

@jfallows
Copy link

jfallows commented Sep 2, 2022

@fmvilas thanks for responding.

It seems all your comments are really focused on HTTP and REST APIs but these are not the target of AsyncAPI (at least not yet). We're trying to get WebSocket (client-server) and broker-based architectures (Kafka, MQTT, AMQP, etc.) right first.

I think you may have misunderstood, my feedback is not specifically focused on HTTP and REST.

My feedback is about the general problem of AsyncAPI publish/subscribe confusion, and getting to the root cause, i.e. that specifying publish or subscribe is indicating the implementation behavior from a certain perspective, triggering the divergence of frontend vs backend because their implementation behaviors differ even though they are interacting within the same logical service contract. So even if we change the naming terms, the frontend vs backend divergence problem persists while the AsyncAPI specification remains too close to the implementation behavior details.

Instead, it would seem to be helpful to define the contract of interaction patterns, independent of frontend or backend perspective, such as:

  • operations input to a service on a channel asking for a certain operation to be performed
    • optional replyTo channel to indicate success or failure (if rejectable)
  • events indicating via a channel a fact that has already occurred, and therefore cannot be rejected
  • callbacks initiating the equivalent of operations but outbound from the service
    • optional replyTo indicate success or failure (if rejectable)

The syntactic specifics above are not as important as the principle of describing what, not how, thus addressing the root cause of the frontend vs backend divergence.

A frontend perspective would clearly publish to the operation channel, and optionally subscribe to the replyTo channel to receive a response, whereas a backend perspective would clearly subscribe to the operation channel, and optionally publish to the replyTo channel to send a response.

A frontend perspective would subscribe to an events channel to receive those events, while a backend perspective would publish those events to the events channel.

A frontend perspective would subscribe to a callback channel, and optionally publish to the callback replyTo channel to send a response, while a backend perspective would publish to a callback channel and optionally subscribe to the replyTo channel to receive a response.

Therefore a single source of truth in AsyncAPI covers both frontend and backend perspectives automatically.

These interaction patterns would seem to apply equally well to WebSocket (client-server) and broker-based architectures (Kafka, MQTT, AMQP, etc.), without creating a frontend vs backend divergence.

There is also the added benefit of fitting nicely with HTTP and REST request-response, but that's for the future so we won't discuss that just yet. 😉

@ivangsa
Copy link

ivangsa commented Oct 21, 2022

hi after some thoughts while preparing to work on some tooling for v3 I have now mixed feelings about the new "channels menu" approach

As I commented early, I thought this was great as it makes posible documenting integration patterns that span more than one application like SAGAs... not possible with v2

but, on the other side, now with v3 message definitions are "not owned" by the providing application but are defined on a common "channels menu" that is shared on the organization and probably there will be way more friction reggarding write access, concurrent edits, pull requests workflows... etc

I believe this could become troublesome very quickly... For an application that just want describe a domain event on a broker topic that probably "it's owned" by this application", to do so on an organization wide shared resource could be like "for publishing a new REST enpoint, need to edit an organization shared openapi.yml"

I'm a bit worried about this.. Have we thought of these implications?

@magicmatatjahu
Copy link
Member

magicmatatjahu commented Oct 21, 2022

@ivangsa Good observation. I myself had a problem with this new approach of describing the messages at the level of channel and not at the level of operation, but I also understand the approach of a channel like a registry, however, there are still quite a few problems here that you described. Since the channel currently does not have any required fields, it would be possible to add the ability to define traits at the channel level, and this would give the user the ability to override the messages for a given channel.

Another problem I see with the current solution - and I didn't participate in the later discussion and found out after the fact what it looks like - that we don't have the ability to define what message(s) on a given operation we are able to send/receive. In the case of brokers the situation is simple, if the operation is "connected" to the broker under a given channel/eventType then it can receive and send the available messages on a given channel (in theory it should, how it is implemented is another thing), but for example in the case of a client-server, e.g. using websocket I have to (in current solution) create two channels with the same address but with a different set of messages, and depending on whether the operation sends or receives (perform logic from server or front client perspective), it's pointed to a different channel. I think that operations should be able to define the message at their level, but it should be optional, and the message itself should be in the registry of available messages of the given channel.

EDIT: I read the previous comments because I didn't remember the discussion, but I still don't understand why we don't have the ability to define message(s) at the operation level. For the case when a channel operates only on one message it is understandable and message at the operation level does not make sense, but if there are 10 or more such messages? Then why do I need operations, if I could describe that the application sends and receives at the level of the channel itself as:

channels:
  someChannel:
    address: ...
    actions: [send, receive]
    messages:
      ...

Really, please explain it to me because I can see the problem (maybe it doesn't exist) and I see that it cause unnecessary redefinition of channels, but with other messages 😅

cc @fmvilas

@jonaslagoni
Copy link
Member

jonaslagoni commented Oct 24, 2022

but, on the other side, now with v3 message definitions are "not owned" by the providing application but are defined on a common "channels menu" that is shared on the organization and probably there will be way more friction reggarding write access, concurrent edits, pull requests workflows... etc

@ivangsa It's not that they are by default not owned by the application. It is that they can* and it all depends on your use case and how you wish to integrate them. It's a pros and cons game as everything else 🙂 You can decide to take that approach and have all messages and channels defined as a menu all applications reference from, or you can have the information owned by the applications (as in version 2, that does not change, only how you define it).

I believe this could become troublesome very quickly... For an application that just want describe a domain event on a broker topic that probably "it's owned" by this application", to do so on an organization wide shared resource could be like "for publishing a new REST enpoint, need to edit an organization shared openapi.yml"

If you go with a menu, It definitely can! In a larger organization, you would need to have processes in place for how global definitions change, but that's governance, right? 🤔 To me this is something you would need in any regard especially if you start to consider versioning, organization-wide design rules, etc.

When it's just the application that defines it, individuals get more control than having to change something in a global repository. But then you might lose reusability.

Really, please explain it to me because I can see the problem (maybe it doesn't exist) and I see that it cause unnecessary redefinition of channels, but with other messages 😅

@magicmatatjahu the request/reply example (ignore the reply keyword 😄) is a pretty nice representation of how messages, channels, and operations work together where channels are always /: https://github.com/asyncapi/spec/blob/edfce73dd90edaa0b720c47f1105039349e3ef9d/examples/kraken-websocket.yml

Do you find this confusing? And if yes, which parts?

From my perspective I quite like this setup, it's clean and clear what messages are in use for which operations. It is quite a change from v2, as you no longer have this all-in-one channel.

@ivangsa
Copy link

ivangsa commented Oct 24, 2022

@jonaslagoni ahh ok, the reference to the channel menu is just a normal reference, you can split into "channel menu" and "application" or use just one file... (or even $ref from the "channel menu" the channel defined on the "application")

thanks for the aclaration... I was thinking that the split as part of the specification.. is just a normal $ref

@magicmatatjahu
Copy link
Member

magicmatatjahu commented Oct 24, 2022

@jonaslagoni I perfectly understand the current status of v3 spea, but as I specified, these revert channels with the same address but with a different messages list (or a single message) causes, in my opinion, a worse of the UX of spec. Take an example where the operation.messages field exists with a definition:

operation.messages defines the list of messages that the operation processes.
The list must be a subset of the list of messages that are defined in the channel defined by the operation.channel field.

then that spec example you gave could look like (I think that operation.channel: null means that the operation sends a response to the asker/sender? Please correct me if I am wrong):

channels:
  root:
    address: /
    messages:
      ping:
        $ref: '#/components/messages/ping'
      pong:
        $ref: '#/components/messages/pong'
      heartbeat:
        $ref: '#/components/messages/heartbeat'
      systemStatus:
        $ref: '#/components/messages/systemStatus'
      subscribe:
        $ref: '#/components/messages/subscribe'
      unsubscribe:
        $ref: '#/components/messages/unsubscribe'
      
  subscriptionStatus:
    address: null
    messages:
      subscriptionStatus:
        $ref: '#/components/messages/subscriptionStatus'
      
operations:
  pingPong:
    action: send
    channel: 
      $ref: '#/channels/root'
    messages:
      ping:
        $ref: '#/messages/ping'
    reply:
      channel: 
        $ref: '#/channels/root'
      messages:
        pong:
          $ref: '#/messages/pong'
  heartbeat:
    action: receive
    channel: 
      $ref: '#/channels/root'
    messages:
      heartbeat:
        $ref: '#/messages/heartbeat'
  systemStatus:
    action: receive
    channel: 
      $ref: '#/channels/root'
    messages:
      systemStatus
        $ref: '#/messages/systemStatus'
  subscribe:
    action: send
    channel: 
      $ref: '#/channels/root'
    messages:
      subscribe:
        $ref: '#/messages/subscribe'
    reply:
      channel: 
        $ref: '#/channels/subscriptionStatus'
  unsubscribe:
    action: send
    channel: 
      $ref: '#/channels/root'
    messages:
      unsubscribe
        $ref: '#/messages/unsubscribe'
    reply:
      channel: 
        $ref: '#/channels/subscriptionStatus'   

channel: $ref: '#/channels/subscriptionStatus' means that operation can operate on all messages defined in channel.

The operation.messages would have the advantage that it would be an optional field, that is, depending on what use case you have or how you want to do it, you can either base the logic on the channels themselves, or add operation.messages as a "filter" for the operation.

I have a huge problem with the examples we have because we only use channel.address, channel.messages and sometimes channel.description. The problem for me is that we forget that the channel itself can also have information like servers, bindings, externalDocs, parameters etc and in the current situation when the channels differ only in the set of messages and the rest is the same (it means that you apply "filter" on which messages given operations can operate, but it's the same channel) we have to copy this data, e.g. (by editing the above mentioned spec):

channels:
  ping:
    address: /{root}
    parameters:
      root: ...
    servers: 
      - $ref: #/servers/someServer1
      - $ref: #/servers/someServer2
    bindings:
      kafka:
        ...
    messages:
      ping:
        $ref: '#/components/messages/ping'
        
  pong:
    address: /{root}
    parameters:
      root: ...
    servers: 
      - $ref: #/servers/someServer1
      - $ref: #/servers/someServer2
    bindings:
      kafka:
        ...
    messages:
      pong:
        $ref: '#/components/messages/pong'

  heartbeat:
    address: /{root}
    parameters:
      root: ...
    servers: 
      - $ref: #/servers/someServer1
      - $ref: #/servers/someServer2
    bindings:
      kafka:
        ...
    messages:
      heartbeat:
        $ref: '#/components/messages/heartbeat'
      
  systemStatus:
    address: /{root}
    parameters:
      root: ...
    servers: 
      - $ref: #/servers/someServer1
      - $ref: #/servers/someServer2
    bindings:
      kafka:
        ...
    messages:
      systemStatus:
        $ref: '#/components/messages/systemStatus'
      
  subscriptionStatus:
    address: null
    messages:
      subscriptionStatus:
        $ref: '#/components/messages/subscriptionStatus'
        
  subscribe:
    address: /{root}
    parameters:
      root: ...
    servers: 
      - $ref: #/servers/someServer1
      - $ref: #/servers/someServer2
    bindings:
      kafka:
        ...
    messages:
      subscribe:
        $ref: '#/components/messages/subscribe'
        
  unsubscribe:
    address: /{root}
    parameters:
      root: ...
    servers: 
      - $ref: #/servers/someServer1
      - $ref: #/servers/someServer2
    bindings:
      kafka:
        ...

Of course, nothing prevents you from defining all these things in components and making refs everywhere, but in this case with the use of operation.messages you would not have as much problem with the "reusability" of the channel and its parts. I see two solutions to the above problem:

  • the possibility of defining traits for channels
  • the possibility of defining operation.messages

I like both options very much and they solve the above problem which I described in 2 different ways, but I would prefer to be able to use two solutions than just channel.traits.

There are, of course, use cases when defining operation.messages will not solve the problem and defining a separate channel with the same address is only a way.

WDYT?

@buehlefs
Copy link

Please also keep in mind that, regardless of what the spec will say a channel or an operation is, people will have their own understandings of these words (especially if they have not read the spec in detail yet). Going against these common understandings will only result in frequent misunderstandings.
An operation is a thing you can do. The documentation for an operation should describe what is the expected result and what needs to be done to call/start the operation (and this information should be contained directly in the operation documentation if possible). In the AsyncAPI context this would be what message needs to be sent to / received from which channel (and what reply can be expected). This would align well with a operation.message keyword.
My intuitive understanding of a message list defined on a channel is that if I wiretap into the channel then I can observe these messages. This does not work if I have to define x channels with the same address as then I no longer have a single source of truth for the complete message list (which also goes against the original channel menu idea to some degree). Also defining these chnnels only because I need to do so to document my operations seems like a very cumbersome workaround at best.
I already wrote a comment outlining a detailed semantic for how a operation.message field could work with channel message lists defined. This comment also describes a way to define the channel message list as "open" as in "open for extension" by operations or applications. This could be a middle ground between requiring all messages to be defined in the channel up front and definign all messages when needed in operations.

@fmvilas
Copy link
Member Author

fmvilas commented Oct 24, 2022

@magicmatatjahu you just gave me an idea and I proposed we apply it (slightly different): #847 (comment).

@leandrorebelo
Copy link

It's an exciting proposal. I'm looking for something like this because We have this situation:

  1. There is a service (Microservice) that reads a Queue (Channel)
  2. We use Command Message Pattern (EIP), so, the Messages have a header called command that informs the service what's handler needs to execute.

In this perspective, I have one Queue (Channel) to many Operations (Commands) and believe that the proposal fits the pattern because I can specify many Operations to a single Channel.

Thanks @fmvilas

@jonaslagoni
Copy link
Member

Just an update based on the issue TLDR. If there are additional changes that have been done for this issue that I have missed feel free to add them 🙂

Remote and Local servers

Is currently not being championed for version 3: #689

Channel Identifiers

Completed

Operations

Completed

Send/Receive

Completed

Optional channels

Completed

More reusability

Completed

That means overall this issue is done and has been implemented both in the parser, JSON Schema documents, and the spec itself 🎉

@github-actions
Copy link

github-actions bot commented Jul 1, 2023

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

@github-actions github-actions bot added the stale label Jul 1, 2023
@fmvilas
Copy link
Member Author

fmvilas commented Jul 6, 2023

It's almost there, dear bot. Don't close it.

@github-actions github-actions bot removed the stale label Jul 7, 2023
Copy link

github-actions bot commented Nov 5, 2023

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💭 Strawman (RFC 0) RFC Stage 0 (See CONTRIBUTING.md)
Projects
None yet
Development

No branches or pull requests