This repository has been archived by the owner on Jul 27, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 120
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
java-openliberty: Add kafka template (#775)
* Add Kafka template * Fix space in directory name * Allow connections to Kafka from host * Switch to provided-scope API dependency; add profiles for host mode Signed-off-by: Scott Kurz <[email protected]> * Bump stack version Co-authored-by: Scott Kurz <[email protected]> Co-authored-by: Ian Partridge <[email protected]> Co-authored-by: Scott Kurz <[email protected]> Co-authored-by: Scott Kurz <[email protected]> Co-authored-by: Kamran Shamsi <[email protected]>
- Loading branch information
5 people
authored
May 1, 2020
1 parent
1ca4940
commit c6e1210
Showing
15 changed files
with
633 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| !.keep | ||
|
|
||
| target/ | ||
|
|
||
| ### STS ### | ||
| .apt_generated | ||
| .classpath | ||
| .factorypath | ||
| .project | ||
| .settings | ||
| .springBeans | ||
| .sts4-cache | ||
|
|
||
| ### IntelliJ IDEA ### | ||
| .idea | ||
| *.iws | ||
| *.iml | ||
| *.ipr | ||
|
|
||
| ### NetBeans ### | ||
| /nbproject/private/ | ||
| /nbbuild/ | ||
| /dist/ | ||
| /nbdist/ | ||
| /.nb-gradle/ | ||
| /build/ | ||
|
|
||
| ### VS Code ### | ||
| .vscode/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Kafka template for Open Liberty | ||
|
|
||
| This template can be used to develop Liberty applications that connect to Kafka by using MicroProfile Reactive messaging. A simple `StarterApplication` is included that enables basic production and consumption of events. | ||
|
|
||
|
|
||
| ## Getting Started with the StarterApplication. | ||
|
|
||
| ### 1. Create a new folder and initialize it using appsody init: | ||
|
|
||
|
|
||
| ``` | ||
| mkdir test-appsody-kafka | ||
| cd test-appsody-kafka | ||
| appsody init java-openliberty kafka | ||
| ``` | ||
|
|
||
| ### 2. Start Kafka and ZooKeeper | ||
|
|
||
| In order to run the `StarterApplication` you must start Kafka and ZooKeeper containers. ZooKeeper is a dependency of Kafka. Use the `docker-compose.yaml` that is provided in the template to start both containers. | ||
|
|
||
|
|
||
| Start docker compose with the following command: | ||
|
|
||
| ```docker-compose up``` | ||
|
|
||
| If you run `docker network list`, you should see a new network with the name of your project directory and the word `_default` appended. For example, `test-appsody-kafka_default`. | ||
|
|
||
| Alternatively, if you want to connect to a Kafka broker elsewhere, edit `src/main/resources/META-INF/microprofile-config.properties` and set the value of the `mp.messaging.connector.liberty-kafka.bootstrap.servers` property to the host and port number of the your broker. | ||
|
|
||
| ### 3. Run the Appsody application in the new network | ||
|
|
||
| Your Appsody application must be run in the same network as Kafka. | ||
|
|
||
| Run the application using the following command: | ||
|
|
||
| ```appsody run --network test-appsody-kafka_default``` | ||
|
|
||
| ### 4. Produce a message to a topic | ||
|
|
||
| Run another container in the same network: | ||
|
|
||
| ```docker run -it --network test-appsody-kafka_default strimzi/kafka:0.16.0-kafka-2.4.0 /bin/bash``` | ||
|
|
||
| The next step is to produce a message. Use the following command to start a Kafka Producer that writes to `incomingTopic1`: | ||
|
|
||
| ```bin/kafka-console-producer.sh --broker-list kafka:9092 --topic incomingTopic1``` | ||
|
|
||
| Enter text at the prompt to produce a message. | ||
|
|
||
| ### 5. Consume a message from a topic | ||
|
|
||
| To view the messages, you can either look at the console log from the Appsody application or you can create a Kafka console consumer using the following command: | ||
|
|
||
| ```bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic incomingTopic1 --from-beginning``` | ||
|
|
||
| ## Deploying to Kubernetes | ||
|
|
||
| When deploying to a Kubernetes environment, you must configure your application to connect to the Kafka broker. You can use the [Strimzi Kafka operator](https://strimzi.io/docs/quickstart/latest/) to deploy a Kafka broker in a Kubernetes cluster. | ||
|
|
||
| To configure the connection, first run the following command: | ||
|
|
||
| ```appsody build``` | ||
|
|
||
| This command generates `app-deploy.yaml` file. | ||
| Edit the file to override the bootstrap server configuration by setting an enviornment variable as follows: | ||
|
|
||
| ``` | ||
| spec: | ||
| env: | ||
| - name: MP_MESSAGING_CONNECTOR_LIBERTY_KAFKA_BOOTSTRAP_SERVERS | ||
| value: <your-kafka-host>:9092 | ||
| ``` | ||
|
|
||
| Then run the following command to deploy your application: | ||
|
|
||
| ``` | ||
| appsody deploy --no-build | ||
| ``` |
34 changes: 34 additions & 0 deletions
34
incubator/java-openliberty/templates/kafka/docker-compose.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| version: '2' | ||
| services: | ||
| zookeeper: | ||
| image: strimzi/kafka:0.17.0-kafka-2.4.0 | ||
| command: [ | ||
| "sh", "-c", | ||
| "bin/zookeeper-server-start.sh config/zookeeper.properties" | ||
| ] | ||
| ports: | ||
| - "2181:2181" | ||
| environment: | ||
| LOG_DIR: /tmp/logs | ||
|
|
||
| kafka: | ||
| image: strimzi/kafka:0.17.0-kafka-2.4.0 | ||
| command: [ | ||
| "sh", "-c", | ||
| "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override listener.security.protocol.map=$${KAFKA_LISTENER_SECURITY_PROTOCOL_MAP} --override inter.broker.listener.name=$${KAFKA_INTER_BROKER_LISTENER_NAME} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}" | ||
| ] | ||
| depends_on: | ||
| - zookeeper | ||
| ports: | ||
| - "9092:9092" | ||
| - "9093:9093" | ||
| expose: | ||
| - "9092" | ||
| - "9093" | ||
| environment: | ||
| LOG_DIR: "/tmp/logs" | ||
| KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9093 | ||
| KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093 | ||
| KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT | ||
| KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE | ||
| KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 |
Oops, something went wrong.