Merge pull request #1 from zendesk/master

Merging upstream master

Author: mavendonovanhubbard

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Maxwell changelog
 
+### [v1.40.0](https://github.com/zendesk/maxwell/releases/tag/v1.40.0)
+
+- add kafka 3.4.0
+- kafka 2.7.0 is now the default kafka library
+- add custom health-check factory jar thing
+
+
+
+_Released 2023-04-02_
+
+### [v1.39.6](https://github.com/zendesk/maxwell/releases/tag/v1.39.6)
+
+- Bugfix issue where SQL query would go missing (#1973)
+- Various parser bugfixes (#1970, #1982, #1987)
+- Fix issue with renaming a primary key column (#1977)
+
+
+
+_Released 2023-03-11_
+
 ### [v1.39.5](https://github.com/zendesk/maxwell/releases/tag/v1.39.5)
 
 - a few parser fixes

Dockerfile

@@ -1,5 +1,5 @@
 FROM maven:3.8-jdk-11 as builder
-ENV MAXWELL_VERSION=1.39.5 KAFKA_VERSION=1.0.0
+ENV MAXWELL_VERSION=1.40.0 KAFKA_VERSION=1.0.0
 
 RUN apt-get update \
     && apt-get -y upgrade \

Makefile

@@ -1,4 +1,4 @@
-KAFKA_VERSION ?= 1.0.0
+KAFKA_VERSION ?= 2.7.0
 KAFKA_PROFILE = kafka-${KAFKA_VERSION}
 export JAVA_TOOL_OPTIONS = -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
 
@@ -16,7 +16,7 @@ clean:
 depclean: clean
 	rm -f $(CLASSPATH)
 
-package: depclean kafka-0.8.2.2 kafka-0.9.0.1 kafka-0.10.0.1 kafka-0.10.2.1 kafka-0.11.0.1 kafka-1.0.0 kafka-2.7.0
+package: depclean kafka-0.8.2.2 kafka-0.9.0.1 kafka-0.10.0.1 kafka-0.10.2.1 kafka-0.11.0.1 kafka-1.0.0 kafka-2.7.0 kafka-3.4.0
 	@# TODO: this is inefficient, we really just want to copy the jars...
 	mvn package -DskipTests=true
 

README.md

@@ -1,44 +1,33 @@
-<div id="maxwell-header">
-</div>
+This is __Maxwell's daemon__, a [change data capture](https://www.confluent.io/blog/how-change-data-capture-works-patterns-solutions-implementation/) application 
+that reads MySQL binlogs and writes data changes as JSON to Kafka, Kinesis, and other streaming platforms.
 
-This is Maxwell's daemon, an application that reads MySQL binlogs and writes
-row updates as JSON to Kafka, Kinesis, or other streaming platforms.  Maxwell has
-low operational overhead, requiring nothing but mysql and a place to write to.
-Its common use cases include ETL, cache building/expiring, metrics collection,
-search indexing and inter-service communication.  Maxwell gives you some of the
-benefits of event sourcing without having to re-architect your entire platform.
 
-<b>Download:</b><br>
-[https://github.com/zendesk/maxwell/releases/download/v1.39.5/maxwell-1.39.5.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.39.5/maxwell-1.39.5.tar.gz)
-<br/>
-<b>Source:</b><br>
-[https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)
-<br clear="all">
 
+[↓ Download](https://github.com/zendesk/maxwell/releases/download/v1.40.0/maxwell-1.40.0.tar.gz) \|
+[⚝ Source / Community](https://github.com/zendesk/maxwell) \|
+[☝ Getting Started](/quickstart) \|
+[☷ Reference](/config)
+
+__What's it for?__
+
+- ETL of all sorts
+- maintaining an audit log of all changes to your database
+- cache building/expiring
+- search indexing 
+- inter-service communication
+
+
+__It goes like this:__
 
 ```
-  mysql> insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem';
-  maxwell: {
+  mysql> update `test`.`maxwell` set mycol = 55, daemon = 'Stanislaw Lem';
+  maxwell -> kafka: 
+  {
     "database": "test",
     "table": "maxwell",
     "type": "insert",
     "ts": 1449786310,
-    "xid": 940752,
-    "commit": true,
-    "data": { "id":1, "daemon": "Stanislaw Lem" }
-  }
-```
-
-```
-  mysql> update test.maxwell set daemon = 'firebus!  firebus!' where id = 1;
-  maxwell: {
-    "database": "test",
-    "table": "maxwell",
-    "type": "update",
-    "ts": 1449786341,
-    "xid": 940786,
-    "commit": true,
-    "data": {"id":1, "daemon": "Firebus!  Firebus!"},
-    "old":  {"daemon": "Stanislaw Lem"}
+    "data": { "id":1, "daemon": "Stanislaw Lem", "mycol": 55 },
+    "old": { "mycol":, 23, "daemon": "what once was" }
   }
 ```

bin/maxwell

@@ -18,7 +18,7 @@ fi
 
 CLASSPATH="$CLASSPATH:$lib_dir/*"
 
-KAFKA_VERSION="1.0.0"
+KAFKA_VERSION="2.7.0"
 
 function use_kafka() {
 	wanted="$1"

docs/docs/config.md

@@ -236,6 +236,7 @@ metrics_datadog_apikey   | STRING | the datadog api key to use when metrics_data
 metrics_datadog_site     | STRING | the site to publish metrics to when metrics_datadog_type = `http` | us
 metrics_datadog_host     | STRING | the host to publish metrics to when metrics_datadog_type = `udp` | localhost
 metrics_datadog_port     | INT | the port to publish metrics to when metrics_datadog_type = `udp` | 8125
+custom_health.factory	| CLASS_NAME                          | fully qualified maxwell health check factory class, see [example](https://github.com/zendesk/maxwell/blob/master/src/example/com/zendesk/maxwell/example/maxwellhealthcheckfactory/CustomMaxwellHealthCheckFactory.java) |
 
 _See also:_ [Monitoring](/monitoring)
 

docs/docs/monitoring.md

@@ -31,11 +31,18 @@ When the HTTP server is enabled the following endpoints are exposed:
 
 | endpoint       | description                                                                    |
 |:---------------|:-------------------------------------------------------------------------------|
-| `/metrics`     | return all metrics as JSON                                                     |
-| `/prometheus`  | return all metrics as Prometheus format                                        |
+| `/metrics`     | GET all metrics as JSON                                                        |
+| `/prometheus`  | GET all metrics as Prometheus format                                           |
 | `/healthcheck` | run Maxwell's healthchecks.  Considered unhealthy if >0 messages have failed in the last 15 minutes. |
 | `/ping`        | a simple ping test, responds with `pong`                                       |
+| `/diagnostics` | for kafka, send a fake message that measures the client to server latency      |
 
+## Custom Health Check
+Similar to the custom producer, developers can provide their own implementation of a health check.
+
+In order to register your custom health check, you must implement the `MaxwellHealthCheckFactory` interface, which is responsible for creating your custom `MaxwellHealthCheck`. Next, set the `custom_health.factory` configuration property to your `MaxwellHealthCheckFactory`'s fully qualified class name. Then add the custom `MaxwellHealthCheckFactory` JAR and all its dependencies to the $MAXWELL_HOME/lib directory.
+
+Custom health check factory and health check examples can be found here: [https://github.com/zendesk/maxwell/tree/master/src/example/com/zendesk/maxwell/example/maxwellhealthcheckfactory](https://github.com/zendesk/maxwell/tree/master/src/example/com/zendesk/maxwell/example/maxwellhealthcheckfactory)
 
 # JMX Configuration
 ***

docs/docs/quickstart.md

@@ -1,13 +1,13 @@
 # Download
 ***
-- Download binary distro: [https://github.com/zendesk/maxwell/releases/download/v1.39.5/maxwell-1.39.5.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.39.5/maxwell-1.39.5.tar.gz)
+- Download binary distro: [https://github.com/zendesk/maxwell/releases/download/v1.40.0/maxwell-1.40.0.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.40.0/maxwell-1.40.0.tar.gz)
 - Sources and bug tracking is available on github: [https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)
 
 **curl**:
 ```
-curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.39.5/maxwell-1.39.5.tar.gz \
+curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.40.0/maxwell-1.40.0.tar.gz \
        | tar zxvf -
-cd maxwell-1.39.5
+cd maxwell-1.40.0
 ```
 
 **docker**:

docs/maxwell_theme/main.html

@@ -1,7 +1,7 @@
 {% extends "base.html" %}
 {%- block content %}
-<div class="col-md-3">{% include "toc.html" %}</div>
-<div class="col-md-9" role="main">
+<div class="col-md-4">{% include "toc.html" %}</div>
+<div class="col-md-8" role="main">
     <div id="{{ page.title }}">
         {% include "content.html" %}
     </div>

pom.xml

@@ -4,7 +4,7 @@
 
   <groupId>com.zendesk</groupId>
   <artifactId>maxwell</artifactId>
-  <version>1.39.5</version>
+  <version>1.40.0</version>
   <packaging>jar</packaging>
 
   <name>maxwell</name>
@@ -123,6 +123,19 @@
         </dependency>
       </dependencies>
     </profile>
+    <profile>
+      <id>kafka-3.4.0</id>
+      <activation>
+        <activeByDefault>true</activeByDefault>
+      </activation>
+      <dependencies>
+        <dependency>
+          <groupId>org.apache.kafka</groupId>
+          <artifactId>kafka-clients</artifactId>
+          <version>3.4.0</version>
+        </dependency>
+      </dependencies>
+    </profile>
   </profiles>
 
   <dependencies>