Merge branch 'release/0.9.81'

aklivity · May 24, 2024 · d7a9265 · d7a9265
2 parents cfea1f0 + 85575e2
commit d7a9265
Show file tree

Hide file tree

Showing 319 changed files with 15,303 additions and 9,941 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,7 +2,23 @@
 
 ## [Unreleased](https://github.com/aklivity/zilla/tree/HEAD)
 
-[Full Changelog](https://github.com/aklivity/zilla/compare/0.9.79...HEAD)
+[Full Changelog](https://github.com/aklivity/zilla/compare/0.9.80...HEAD)
+
+**Implemented enhancements:**
+
+- Improve Starting Zilla with the CLI [\#1016](https://github.com/aklivity/zilla/issues/1016)
+- Generate `zilla dump` packet captures in timestamp order including across workers [\#959](https://github.com/aklivity/zilla/issues/959)
+- Split protocol testing into separate ITs for `zilla dump` command [\#958](https://github.com/aklivity/zilla/issues/958)
+- Add zilla context to MQTT consumer groups [\#886](https://github.com/aklivity/zilla/issues/886)
+
+**Fixed bugs:**
+
+- Telemetry attribute service.name doesn't get sent correctly [\#1007](https://github.com/aklivity/zilla/issues/1007)
+- Streampay `zilla` instance crashes while trying to access  `https://localhost:9090` [\#975](https://github.com/aklivity/zilla/issues/975)
+
+## [0.9.80](https://github.com/aklivity/zilla/tree/0.9.80) (2024-05-20)
+
+[Full Changelog](https://github.com/aklivity/zilla/compare/0.9.79...0.9.80)
 
 **Breaking changes:**
 
@@ -23,6 +39,7 @@
 - Resiliently handle `apicurio` catalog unreachable [\#938](https://github.com/aklivity/zilla/issues/938)
 - Resiliently handle `karapace` catalog unreachable [\#937](https://github.com/aklivity/zilla/issues/937)
 - Support local `zilla` installation on MacOS via `homebrew` [\#680](https://github.com/aklivity/zilla/issues/680)
+- Update bug report template [\#820](https://github.com/aklivity/zilla/pull/820) ([vordimous](https://github.com/vordimous))
 
 **Fixed bugs:**
 
@@ -35,6 +52,46 @@
 - Using the `grpc.kafka.proxy` example setup, Zilla will periodically not deliver the message to the gRPC service defined in the `kafka-grpc` binding [\#881](https://github.com/aklivity/zilla/issues/881)
 - Flaky kafka-grpc test [\#768](https://github.com/aklivity/zilla/issues/768)
 
+**Merged pull requests:**
+
+- Handle & calculate complex schema padding [\#1038](https://github.com/aklivity/zilla/pull/1038) ([aDaemonThread](https://github.com/aDaemonThread))
+- Fix typo to send abort on abort instead of end [\#1036](https://github.com/aklivity/zilla/pull/1036) ([akrambek](https://github.com/akrambek))
+- Bump junit.version from 5.10.1 to 5.10.2 [\#1032](https://github.com/aklivity/zilla/pull/1032) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Honor MQTT clean start at QoS2 produce [\#1031](https://github.com/aklivity/zilla/pull/1031) ([bmaidics](https://github.com/bmaidics))
+- Use flyweight fields instead of class fields for control [\#1030](https://github.com/aklivity/zilla/pull/1030) ([akrambek](https://github.com/akrambek))
+- catalog:apicurio - unify caching across workers to maximize cache hits [\#1027](https://github.com/aklivity/zilla/pull/1027) ([aDaemonThread](https://github.com/aDaemonThread))
+- Use binding id instead of route Id for resolved Id [\#1026](https://github.com/aklivity/zilla/pull/1026) ([akrambek](https://github.com/akrambek))
+- MQTT clients access log implementation [\#1023](https://github.com/aklivity/zilla/pull/1023) ([aDaemonThread](https://github.com/aDaemonThread))
+- Unsubscribe on partition reassignment [\#1021](https://github.com/aklivity/zilla/pull/1021) ([akrambek](https://github.com/akrambek))
+- Bump commons-cli:commons-cli from 1.6.0 to 1.7.0 [\#1020](https://github.com/aklivity/zilla/pull/1020) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Increase mqtt client id limit to 256 [\#1015](https://github.com/aklivity/zilla/pull/1015) ([bmaidics](https://github.com/bmaidics))
+- Generate correct crc32c value for the messages with different produceId [\#1011](https://github.com/aklivity/zilla/pull/1011) ([akrambek](https://github.com/akrambek))
+- Remove generate command [\#1010](https://github.com/aklivity/zilla/pull/1010) ([attilakreiner](https://github.com/attilakreiner))
+- Support configuration of timestamps in zilla transport for k3po [\#1009](https://github.com/aklivity/zilla/pull/1009) ([jfallows](https://github.com/jfallows))
+- Support multiple specs in openapi-asyncapi binding [\#1008](https://github.com/aklivity/zilla/pull/1008) ([bmaidics](https://github.com/bmaidics))
+- Support multiple specs in openapi binding [\#1005](https://github.com/aklivity/zilla/pull/1005) ([bmaidics](https://github.com/bmaidics))
+- Fix secure http detection in OpenAPI [\#1002](https://github.com/aklivity/zilla/pull/1002) ([bmaidics](https://github.com/bmaidics))
+- Enhancing validation for openapi and asyncapi bindings [\#1001](https://github.com/aklivity/zilla/pull/1001) ([aDaemonThread](https://github.com/aDaemonThread))
+- Support asyncapi mqtt streetlights mapping to kafka streetlights [\#997](https://github.com/aklivity/zilla/pull/997) ([bmaidics](https://github.com/bmaidics))
+- Bump ubuntu from jammy-20240416 to jammy-20240427 in /cloud/docker-image/src/main/docker [\#996](https://github.com/aklivity/zilla/pull/996) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump org.bitbucket.b\_c:jose4j from 0.9.3 to 0.9.6 [\#995](https://github.com/aklivity/zilla/pull/995) ([dependabot[bot]](https://github.com/apps/dependabot))
+- MQTT Websocket bugfix [\#993](https://github.com/aklivity/zilla/pull/993) ([bmaidics](https://github.com/bmaidics))
+- Add MQTT client authentication [\#992](https://github.com/aklivity/zilla/pull/992) ([bmaidics](https://github.com/bmaidics))
+- `tls`  `client/server/echo` handshake benchmark  [\#990](https://github.com/aklivity/zilla/pull/990) ([akrambek](https://github.com/akrambek))
+- Bump ubuntu from jammy-20240111 to jammy-20240416 in /cloud/docker-image/src/main/docker [\#987](https://github.com/aklivity/zilla/pull/987) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Bump alpine from 3.19.0 to 3.19.1 in /cloud/docker-image/src/main/docker [\#986](https://github.com/aklivity/zilla/pull/986) ([dependabot[bot]](https://github.com/apps/dependabot))
+- Update Java build matrix [\#983](https://github.com/aklivity/zilla/pull/983) ([jfallows](https://github.com/jfallows))
+- Support multiple specs in asyncapi binding [\#982](https://github.com/aklivity/zilla/pull/982) ([bmaidics](https://github.com/bmaidics))
+- Remove event script in favor of handshake script [\#981](https://github.com/aklivity/zilla/pull/981) ([attilakreiner](https://github.com/attilakreiner))
+- `echo` `server` handshake benchmark [\#980](https://github.com/aklivity/zilla/pull/980) ([akrambek](https://github.com/akrambek))
+- MqttKafka publish intern fix [\#979](https://github.com/aklivity/zilla/pull/979) ([bmaidics](https://github.com/bmaidics))
+- Fix multiple exporters issue [\#978](https://github.com/aklivity/zilla/pull/978) ([attilakreiner](https://github.com/attilakreiner))
+- unify caching across workers to maximize cache hits [\#977](https://github.com/aklivity/zilla/pull/977) ([aDaemonThread](https://github.com/aDaemonThread))
+- Support reading empty file payload [\#976](https://github.com/aklivity/zilla/pull/976) ([jfallows](https://github.com/jfallows))
+- Use format to construct get openapi operation for async rquest [\#967](https://github.com/aklivity/zilla/pull/967) ([akrambek](https://github.com/akrambek))
+- binding config schema validation for unused properties [\#929](https://github.com/aklivity/zilla/pull/929) ([aDaemonThread](https://github.com/aDaemonThread))
+- Bump org.agrona:agrona from 1.6.0 to 1.21.1 [\#890](https://github.com/aklivity/zilla/pull/890) ([dependabot[bot]](https://github.com/apps/dependabot))
+
 ## [0.9.79](https://github.com/aklivity/zilla/tree/0.9.79) (2024-04-22)
 
 [Full Changelog](https://github.com/aklivity/zilla/compare/0.9.78...0.9.79)

diff --git a/build/flyweight-maven-plugin/pom.xml b/build/flyweight-maven-plugin/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>build</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/build/pom.xml b/build/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>zilla</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/cloud/docker-image/pom.xml b/cloud/docker-image/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>cloud</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/cloud/helm-chart/pom.xml b/cloud/helm-chart/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>cloud</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/cloud/pom.xml b/cloud/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>zilla</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/conf/pom.xml b/conf/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>zilla</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/incubator/binding-amqp.spec/pom.xml b/incubator/binding-amqp.spec/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>incubator</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/incubator/binding-amqp/pom.xml b/incubator/binding-amqp/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>incubator</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/incubator/catalog-filesystem.spec/pom.xml b/incubator/catalog-filesystem.spec/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>incubator</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/incubator/catalog-filesystem/pom.xml b/incubator/catalog-filesystem/pom.xml
@@ -6,7 +6,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>incubator</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 

diff --git a/incubator/command-dump/pom.xml b/incubator/command-dump/pom.xml
@@ -8,7 +8,7 @@
   <parent>
     <groupId>io.aklivity.zilla</groupId>
     <artifactId>incubator</artifactId>
-    <version>0.9.80</version>
+    <version>0.9.81</version>
     <relativePath>../pom.xml</relativePath>
   </parent>
 
@@ -43,16 +43,32 @@
       <version>${project.version}</version>
       <scope>provided</scope>
     </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>engine</artifactId>
+      <type>test-jar</type>
+      <version>${project.version}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>junit</groupId>
+      <artifactId>junit</artifactId>
+      <scope>test</scope>
+    </dependency>
     <dependency>
       <groupId>${project.groupId}</groupId>
       <artifactId>command</artifactId>
       <version>${project.version}</version>
       <scope>provided</scope>
     </dependency>
     <dependency>
-      <groupId>io.aklivity.zilla</groupId>
-      <artifactId>binding-proxy.spec</artifactId>
-      <version>${project.version}</version>
+      <groupId>org.kaazing</groupId>
+      <artifactId>k3po.junit</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.kaazing</groupId>
+      <artifactId>k3po.lang</artifactId>
       <scope>test</scope>
     </dependency>
     <dependency>
@@ -99,25 +115,26 @@
     </dependency>
     <dependency>
       <groupId>io.aklivity.zilla</groupId>
-      <artifactId>binding-amqp.spec</artifactId>
+      <artifactId>binding-proxy.spec</artifactId>
       <version>${project.version}</version>
       <scope>test</scope>
     </dependency>
     <dependency>
-      <groupId>org.junit.jupiter</groupId>
-      <artifactId>junit-jupiter-engine</artifactId>
+      <groupId>io.aklivity.zilla</groupId>
+      <artifactId>binding-amqp.spec</artifactId>
+      <version>${project.version}</version>
       <scope>test</scope>
     </dependency>
     <dependency>
-      <groupId>org.testcontainers</groupId>
-      <artifactId>testcontainers</artifactId>
-      <version>1.19.3</version>
+      <groupId>io.aklivity.zilla</groupId>
+      <artifactId>binding-tls.spec</artifactId>
+      <version>${project.version}</version>
       <scope>test</scope>
     </dependency>
     <dependency>
       <groupId>org.testcontainers</groupId>
-      <artifactId>junit-jupiter</artifactId>
-      <version>1.19.3</version>
+      <artifactId>testcontainers</artifactId>
+      <version>1.19.7</version>
       <scope>test</scope>
     </dependency>
     <dependency>
@@ -143,9 +160,9 @@
         <artifactId>license-maven-plugin</artifactId>
         <configuration>
           <excludes>
-            <exclude>src/test/resources/io/aklivity/zilla/runtime/command/dump/internal/airline/*.pcap</exclude>
-            <exclude>src/test/resources/io/aklivity/zilla/runtime/command/dump/internal/airline/*.txt</exclude>
-            <exclude>src/test/resources/io/aklivity/zilla/runtime/command/dump/internal/airline/engine/*</exclude>
+            <exclude>src/test/resources/io/aklivity/zilla/runtime/command/dump/internal/*.txt</exclude>
+            <exclude>**/keys</exclude>
+            <exclude>**/trust</exclude>
           </excludes>
         </configuration>
       </plugin>
@@ -164,6 +181,23 @@
         <groupId>org.moditect</groupId>
         <artifactId>moditect-maven-plugin</artifactId>
       </plugin>
+      <plugin>
+        <groupId>org.kaazing</groupId>
+        <artifactId>k3po-maven-plugin</artifactId>
+        <dependencies>
+          <dependency>
+            <groupId>${project.groupId}</groupId>
+            <artifactId>engine</artifactId>
+            <version>${project.version}</version>
+            <type>test-jar</type>
+          </dependency>
+          <dependency>
+            <groupId>${project.groupId}</groupId>
+            <artifactId>engine</artifactId>
+            <version>${project.version}</version>
+          </dependency>
+        </dependencies>
+      </plugin>
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-failsafe-plugin</artifactId>
@@ -227,6 +261,9 @@
           <include>**/zilla.lua</include>
         </includes>
       </resource>
+      <resource>
+        <directory>src/main/scripts</directory>
+      </resource>
     </resources>
   </build>
 </project>
diff --git a/...c/main/java/io/aklivity/zilla/runtime/command/dump/internal/airline/ZillaDumpCommand.java b/...c/main/java/io/aklivity/zilla/runtime/command/dump/internal/airline/ZillaDumpCommand.java
@@ -191,7 +191,7 @@ public final class ZillaDumpCommand extends ZillaCommand
         hidden = true)
     public boolean exceptions;
 
-    boolean continuous = true;
+    public boolean continuous = true;
 
     private final FrameFW frameRO = new FrameFW();
     private final BeginFW beginRO = new BeginFW();
@@ -217,6 +217,8 @@ public final class ZillaDumpCommand extends ZillaCommand
     private final TcpHeaderFW.Builder tcpHeaderRW = new TcpHeaderFW.Builder();
     private final MutableDirectBuffer patchBuffer;
     private final MutableDirectBuffer writeBuffer;
+    private int minWorkerIndex;
+    private long minTimeStamp;
 
     private Path directory;
 
@@ -332,11 +334,26 @@ public void run()
                 do
                 {
                     workCount = 0;
-                    for (int i = 0; i < streamBufferCount; i++)
+                    minWorkerIndex = 0;
+                    minTimeStamp = Long.MAX_VALUE;
+                    int framesPeeked = 0;
+                    for (int j = 0; j < streamBufferCount; j++)
                     {
-                        final RingBufferSpy streamBuffer = streamBuffers[i];
-                        MessagePredicate spyHandler = dumpHandlers[i]::handleFrame;
-                        workCount += streamBuffer.spy(spyHandler, 1);
+                        final int workerIndex = j;
+                        framesPeeked += streamBuffers[workerIndex].peek((m, b, i, l) ->
+                        {
+                            frameRO.wrap(b, i, i + l);
+                            if (frameRO.timestamp() < minTimeStamp)
+                            {
+                                minTimeStamp = frameRO.timestamp();
+                                minWorkerIndex = workerIndex;
+                            }
+                        });
+                    }
+                    if (framesPeeked > 0)
+                    {
+                        MessagePredicate spyHandler = dumpHandlers[minWorkerIndex]::handleFrame;
+                        workCount += streamBuffers[minWorkerIndex].spy(spyHandler, 1);
                     }
                     idleStrategy.idle(workCount);
                 } while (workCount != exitWorkCount);

diff --git a/...va/io/aklivity/zilla/runtime/command/dump/internal/airline/spy/OneToOneRingBufferSpy.java b/...va/io/aklivity/zilla/runtime/command/dump/internal/airline/spy/OneToOneRingBufferSpy.java
@@ -30,6 +30,7 @@
 import org.agrona.DirectBuffer;
 import org.agrona.concurrent.AtomicBuffer;
 
+import io.aklivity.zilla.runtime.engine.binding.function.MessageConsumer;
 import io.aklivity.zilla.runtime.engine.binding.function.MessagePredicate;
 
 public class OneToOneRingBufferSpy implements RingBufferSpy
@@ -146,4 +147,26 @@ public int spy(
 
         return messagesRead;
     }
+
+    @Override
+    public int peek(
+        final MessageConsumer handler)
+    {
+        final AtomicBuffer buffer = this.buffer;
+        final long head = spyPosition.get();
+        final int capacity = this.capacity;
+        final int headIndex = (int)head & (capacity - 1);
+        final int recordLength = buffer.getIntVolatile(lengthOffset(headIndex));
+        int messagesPeeked = 0;
+        if (recordLength > 0)
+        {
+            final int messageTypeId = buffer.getInt(typeOffset(headIndex));
+            if (PADDING_MSG_TYPE_ID != messageTypeId)
+            {
+                handler.accept(messageTypeId, buffer, headIndex + HEADER_LENGTH, recordLength - HEADER_LENGTH);
+                messagesPeeked = 1;
+            }
+        }
+        return messagesPeeked;
+    }
 }
diff --git a/.../main/java/io/aklivity/zilla/runtime/command/dump/internal/airline/spy/RingBufferSpy.java b/.../main/java/io/aklivity/zilla/runtime/command/dump/internal/airline/spy/RingBufferSpy.java
@@ -16,6 +16,7 @@
 
 import org.agrona.DirectBuffer;
 
+import io.aklivity.zilla.runtime.engine.binding.function.MessageConsumer;
 import io.aklivity.zilla.runtime.engine.binding.function.MessagePredicate;
 
 public interface RingBufferSpy
@@ -31,6 +32,7 @@ enum SpyPosition
 
     int spy(MessagePredicate handler);
     int spy(MessagePredicate handler, int messageCountLimit);
+    int peek(MessageConsumer handler);
 
     long producerPosition();
     long consumerPosition();