Issue with installing Ansible RM to integrate with Kafka in ICP

[razikhussein]

The configuration for Ansible RM only takes in kafka hostname.
And it assumes the port is fixed at 9092

When installed on ICP, there are 3 instance of Kafka, and the port exposed are different.
How can we make the Ansible RM work with thus?

[joschkappel]

ansible-RM can be configured using the config.yml settings.
To modify kafka ports change this setting:

driver:
  properties:
    - responseKafkaConnectionUrl: "kafka:9092"

The host name mappgin can be set in the docker-compose file:

services:
  alm-ansible-rm:
         extra_hosts:
           - "kafka:192.168.61.64"

However you should not connect to individual kafka container replicas on ALM on ICP. ALM defines a kafka service (default port is 9092) which should be used.
I expect a missing ingress is blocking access to this service from outside the helm release.

[razikhussein]

I have an Ingress created as below:

[root@icpdevmaster01 ingress]# kubectl get ingress -n time-alm
NAME HOSTS ADDRESS PORTS AGE
ishtar-ingress * 10.210.116.80 80 15d
kafka-ingress * 10.210.116.80 80 101s
nimrod-ingress * 10.210.116.80 80 15d

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
labels:
app: kafka
name: kafka-ingress
spec:
rules:

• http:
  paths:
  • path: /kafka
    backend:
    serviceName: alm-kafka
    servicePort: 9092

Is this correct?
Should I be using the ip address 10.210.116.80 for kafka_host?

[joschkappel]

I understand that kube ingress objects route HTTP(S) traffic only.
It should be sufficient to define the kafka service of type NodePort or LoadBalancer and assign he node IP.
This IP and the NodePort port should then be used in the rm settings

Maybe this thread helps: kafka issue on kubernetes
or here:
external communication with kafka on k8s

[Gmikeit]

Elsewhere we define a comma separated list of master node IPs and ports representing the Kafka bootstrap.server
Can this be configured in alm-ansible-rm ?
@razikhussein fyi

[joschkappel]

Yes in the config.yml you can set for example:

driver:
  properties:
    - responseKafkaConnectionUrl: "192.168.63.179:9092,192.168.63.179:9093,192.168.61.239:9094"

[razikhussein]

Hi Jochen,

When I do a simple grep, there are 4 files where this 9092 is mentioned (as below). You mentioned above about config.yml only (to change the port to other than default 9092), should be leave the others as it is or should we change that as well)?

/ansible-adaptor/config.yml
/ansible-adaptor/swagger_server/controllers/ans_kafka_topic.py
/helm/osslm-ansible-resource-manager/templates/lm-kafka-service.yaml
/helm/osslm-ansible-resource-manager/values.yaml

[razikhussein]

Also if I update config.yml as below

driver:
properties:
- responseKafkaConnectionUrl: "192.168.63.179:9092,192.168.63.179:9093,192.168.61.239:9094"

How would the alm-ansible-rm-docker-compose.yml for below looks like?

 extra_hosts:
   - "kafka:10.210.116.53"

[Gmikeit]

setting,
- responseKafkaConnectionUrl: "192.168.63.179:9092,192.168.63.179:9093,192.168.61.239:9094"
... results in messages like this:
[2019-08-20 03:32:41,408] {ans_kafka:43} ERROR - could not create kafka topic at 10.210.116.60:30090,10.210.116.61:30638,10.210.116.62:32386

[joschkappel]

We successfully tested here this setup (ALM 2.02, ARM latest from master):

On ICP we applied:

helm upgrade alm --namespace lifecycle-manager charts/ibm-alm-prod-2.0.0.tgz --values=expose_kafka.yaml --values=pak_extensions/prereqs/custom-secrets.yaml --tls

set advertised listener as master node:

global:
  environmentSize: size0
  image:
    repository: mycluster.icp:8500/lifecycle-manager
  kafka:
    clusterSize: 1
kafka:
  advertisedListeners:
    - <masterNode>

This gives you one kafka entrypoint. Load balancing is done by ICP.
On ARM side just use kafak_ip:port in the docker-compose file for KAFKA_URL.

[Gmikeit]

We successfully tested here this setup (ALM 2.02, ARM latest from master):

On ICP we applied:

helm upgrade alm --namespace lifecycle-manager charts/ibm-alm-prod-2.0.0.tgz --values=expose_kafka.yaml --values=pak_extensions/prereqs/custom-secrets.yaml --tls

set advertised listener as master node:

global:
  environmentSize: size0
  image:
    repository: mycluster.icp:8500/lifecycle-manager
  kafka:
    clusterSize: 1
kafka:
  advertisedListeners:
    - <masterNode>

This gives you one kafka entrypoint. Load balancing is done by ICP.
On ARM side just use kafak_ip:port in the docker-compose file for KAFKA_URL.

ALM 2.02 is a typo, should read ALM 2.0.0