ERROR: Service 'kafka' failed to build: #565

ckvtvm-arch · 2020-01-22T01:50:57Z

Step 11/14 : RUN apk add --no-cache bash curl jq docker && chmod a+x /tmp/.sh && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKA_HOME} && rm /tmp/ && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk && rm glibc-${GLIBC_VERSION}.apk
---> Running in b715c5d84399
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz Installing jq (1.6-r0)
Executing busybox-1.29.3-r10.trigger
OK: 349 MiB in 74 packages
/bin/sh: illegal option -
ERROR: Service 'kafka' failed to build: The command '/bin/sh -c apk add --no-cache bash curl jq docker && chmod a+x /tmp/.sh && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKA_HOME} && rm /tmp/ && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk && rm glibc-${GLIBC_VERSION}.apk' returned a non-zero code: 2

bellhop · 2020-01-22T18:22:34Z

Looks like this issue is there is no support for TLS1.2. If you add --no-check-certificate to line 35 of the Dockerfile it will run the wget command and allow the self signed cert.

ckvtvm-arch · 2020-01-22T19:55:37Z

Step 11/14 : RUN apk add --no-cache bash curl jq docker && chmod a+x /tmp/.sh && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKA_HOME} && rm /tmp/ && wget --no-check-certificate https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk && rm glibc-${GLIBC_VERSION}.apk
---> Running in f0ef042bb572
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz Installing jq (1.6-r0)
Executing busybox-1.29.3-r10.trigger
OK: 349 MiB in 74 packages
: not foundad-kafka.sh: line 1:
: not foundad-kafka.sh: line 3:
': No such file or directoryce: line 5: can't open '/usr/bin/versions.sh
ERROR: Service 'kafka' failed to build: The command '/bin/sh -c apk add --no-cache bash curl jq docker && chmod a+x /tmp/.sh && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKA_HOME} && rm /tmp/ && wget --no-check-certificate https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk && rm glibc-${GLIBC_VERSION}.apk' returned a non-zero code: 2

ckvtvm-arch · 2020-01-22T19:56:20Z

I think it is the issue with download-kafka.sh as below:

#!/bin/sh -e

shellcheck disable=SC1091

source "/usr/bin/versions.sh"

FILENAME="kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"

url=$(curl --stderr /dev/null "https://www.apache.org/dyn/closer.cgi?path=/kafka/${KAFKA_VERSION}/${FILENAME}&as_json=1" | jq -r '"(.preferred)(.path_info)"')

Test to see if the suggested mirror has this version, currently pre 2.1.1 versions

do not appear to be actively mirrored. This may also be useful if closer.cgi is down.

if [[ ! $(curl -s -f -I "${url}") ]]; then
echo "Mirror does not have desired version, downloading direct from Apache"
url="https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${FILENAME}"
fi

echo "Downloading Kafka from $url"
wget "${url}" -O "/tmp/${FILENAME}"

OneCricketeer · 2020-01-22T21:38:57Z

Do you need to build it? docker pull wurstmeister/kafka works fine

dgecawich · 2020-01-28T03:11:15Z

I am encountering the same issue building kafka using the latest docker-compose.yml. Error is identical:

**ERROR**: Service 'kafka' failed to build: The command '/bin/sh -c apk add --no-cache bash curl jq docker && chmod a+x /tmp/*.sh && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin && sync && /tmp/download-kafka.sh && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKA_HOME} && rm /tmp/* && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk && rm glibc-${GLIBC_VERSION}.apk' returned a non-zero code: 2

Workaround: replace docker-compose.yml as follows to pull an older binary docker image rather than build, as follows:

version: '2'

services:

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    expose:
    - "2181"

  kafka:
    image: wurstmeister/kafka:2.11-2.0.0
    depends_on:
    - zookeeper
    ports:
    - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

OneCricketeer · 2020-01-28T03:30:59Z

Again, why do you need to build? And you can use latest tag, no need to use older versions

ckvtvm-arch · 2020-01-28T03:35:28Z

We are building from docker file as we are not able to successfully deploy the already build image in other infrastructure such as openshift, AWS etc Sent from Yahoo Mail for iPhone On Monday, January 27, 2020, 10:31 PM, Jordan Moore <[email protected]> wrote: Again, why do you need to build? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

OneCricketeer · 2020-01-28T03:36:24Z

And what's the error when you do deploy to those environments with existing images?

OneCricketeer · 2020-01-28T03:40:16Z

Regarding openshift, see #403

Or just use Strimzi / AMQ Streams

dgecawich · 2020-01-28T03:52:08Z

For production, the preferred security policy is to stay as close to the official linux (e.g. alpine) base images, add-on services from the base image installed via OS and official download site binaries from Apache Kafka. The build provides substantially more transparency and security verification, than a prebaked image. It also provides a mechanism to extend and customize Kafka. Once the build is completed, the ingesting docker compase.yml may very well end up using the binary docker image after successfully running the build.

ckvtvm-arch · 2020-01-28T03:58:04Z

I think it was always giving issue around zookeeper connect in various images from dockerhub Sent from Yahoo Mail for iPhone On Monday, January 27, 2020, 10:52 PM, David Gecawich <[email protected]> wrote: For production, the preferred security policy is to stay as close to the official linux (e.g. alpine) base images, add-on services from the base image installed via OS and official download site binaries from Apache Kafka. The build provide the necessary transparency and security verification. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

OneCricketeer · 2020-01-28T04:21:21Z

stay as close to the official linux (e.g. alpine) base images, add-on services from the base image installed via OS and official download site binaries from Apache Kafka

That's exactly what the Dockerfile here does... Re-downloading the (now missing) Kafka binaries from the Apache archives don't make the build more secure.

If you were to change the base image, then I'd accept your argument.

provides a mechanism to extend and customize Kafka

Not sure what you would extend that isn't already provided. Plus, FROM wurstmeister/kafka should be good enough for that use case.

Once the build is completed, the ingesting docker compase.yml may very well end up using the binary docker image after successfully running the build

Sure, thats just typical usage of Docker Compose.

OneCricketeer · 2020-01-28T04:22:45Z

I think it was always giving issue around zookeeper connect

You'll need to be more specific about that. Sounds like you might have networking issues, which are not a problem with this container image.

ckvtvm-arch · 2020-01-28T04:36:25Z

Does this image have both Kafka and zoo keeper ? It may not be a network issue as I am deploying this within my project space. Let me send the exact message tomorrow from office Sent from Yahoo Mail for iPhone On Monday, January 27, 2020, 11:22 PM, Jordan Moore <[email protected]> wrote: I think it was always giving issue around zookeeper connect You'll need to be more specific about that. Sounds like you might have networking issues, which are not a problem with this container image. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

OneCricketeer · 2020-01-28T04:38:44Z

Does this image have both Kafka and zoo keeper ?

wurstmeister/kafka and wurstmeister/zookeeper are separate Docker images that should be ran separately.

Again regarding Openshift, plenty of external resources on doing just that (which actually do not use this image)

https://developers.redhat.com/blog/2018/10/29/how-to-run-kafka-on-openshift-the-enterprise-kubernetes-with-amq-streams/

dgecawich · 2020-01-28T04:46:11Z

#565 (comment)

OneCricketeer · 2020-01-28T04:49:44Z

I read the comment, yes. Sorry, what about it?

SCALA_VERSION and KAFKA_VERSION are completely overridable, by the way.

The ultimate reason you're getting build errors is because Apache servers depend on mirrors to host older binaries. They're not all available from the main Apache download site

dgecawich · 2020-01-28T17:24:09Z

@Cricket007 Thank you for the response, which clarified the cause of the issue and workaround/soluition to change SCALA_VERSION and KAFKA_VERSION to a version that is available on the download mirror and re-try. +1 for that response.

ckvtvm-arch · 2020-01-28T18:02:23Z

Does this image have both Kafka and zoo keeper ?

wurstmeister/kafka and wurstmeister/zookeeper are separate Docker images that should be ran separately.

Again regarding Openshift, plenty of external resources on doing just that (which actually do not use this image)

https://developers.redhat.com/blog/2018/10/29/how-to-run-kafka-on-openshift-the-enterprise-kubernetes-with-amq-streams/

ckvtvm-arch · 2020-01-28T18:06:58Z

ERROR: missing mandatory config: KAFKA_ZOOKEEPER_CONNECT after deploying the kafka image while zookeeper image deployment shows as | --> Scaling zookeeper-wurstmeister-1 to 1

As the openshift methods require admin level access and minishift is not something we prefer, hence try deploying the images .

I havent tried re-buiilding from dockerfile with the latest changes, i will update you as soon as i complete.

OneCricketeer · 2020-01-28T18:10:40Z

minishift is not something we prefer

I never mentioned using minishift. The link only shows it to show the application deployment lifecycles.

missing mandatory config: KAFKA_ZOOKEEPER_CONNECT

Sounds like you have not set the correct environment variables onto the container from your k8s YAML spec?

OneCricketeer · 2020-01-28T18:11:34Z

Also relevant - https://strimzi.io/quickstarts/okd/

ckvtvm-arch · 2020-01-28T18:15:52Z

minishift is not something we prefer

I never mentioned using minishift. The link only shows it to show the application deployment lifecycles.

missing mandatory config: KAFKA_ZOOKEEPER_CONNECT

Sounds like you have not set the correct environment variables onto the container from your k8s YAML spec? - I am deploying the complete image ?

ckvtvm-arch · 2020-01-28T18:17:20Z

I am deploying the complete image from dockerhub, do i need K8s Yaml spec ? .
Wurstmeister/kafka latest 9a5842c217a8
wurstmeister/zookeeper latest 3f43f72cb283

Also i dont have admin access at cluster level and has access only at namespace / project space level as a developer to try https://strimzi.io/quickstarts/okd/

ckvtvm-arch · 2020-01-28T18:20:27Z

let me try this - https://github.com/engapa/kafka-k8s-openshift/tree/master/openshift

ccarv · 2020-01-30T14:46:44Z

I am also experiencing an issue when trying to run docker-compose up. I tried adding the --no-check-certificate flag to wget but that does not change the outcome. I am still getting /bin/sh: illegal option - error.

I used @dgecawich recommendation and it started just fine. Although, I would like to use the latest tag.

OneCricketeer · 2020-01-30T14:48:53Z

I am also experiencing an issue when trying to run docker-compose up.

Remove build: . and add image: wurstmeister/kafka:latest

Problem should be solved. Again, I'm finding it unlikely y'all truly need to (re)build the image

ccarv · 2020-01-30T14:49:53Z

Will give that a go. Thank you.

costa92 · 2020-03-03T07:55:31Z

Dockerfile add
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories

OneCricketeer · 2020-03-03T09:01:18Z

@costa92 Thanks but please explain your use case for rebuilding the container

ckvtvm-arch · 2020-03-03T14:01:20Z

Whenever I try to deploy the image in Openshift in my org , my account is getting locked . I am guessing it could be due to in build configurations . Anyway I have requested for my own storage which I will connect to my Openshift Instance in my org and will let you know how it goes ! Sent from Yahoo Mail for iPhone On Thursday, January 30, 2020, 9:49 AM, Chris <[email protected]> wrote: Will give that a go. Thank you. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

costa92 · 2020-03-05T04:02:26Z

@Cricket007 docker image system basic extended source

OneCricketeer · 2020-03-05T04:44:11Z

@costa92 sorry, not sure I understand.

Can you not do FROM wurstmeister/kafka in your own image?

ckvtvm-arch · 2020-05-04T20:37:14Z

It looks like we need cluster level access to deploy this image wherein i have only access at the namespace level. hence I am requesting my OCP administrator to deploy the Red hat AMQ in openshift rather than working on opensource images.

OneCricketeer · 2020-05-04T22:01:16Z

deploy the Red hat AMQ in openshift rather than working on opensource images

Seems like a good idea. Or Strimzi

ckvtvm-arch · 2020-05-04T22:28:59Z

Yes,I think Red hat AMQ is based on Strimzi Sent from Yahoo Mail for iPhone On Monday, May 4, 2020, 6:01 PM, Jordan Moore <[email protected]> wrote: deploy the Red hat AMQ in openshift rather than working on opensource images Seems like a good idea. Or Strimzi — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

ckvtvm-arch closed this as completed Jan 28, 2020

ckvtvm-arch reopened this Jan 28, 2020

ckvtvm-arch closed this as completed May 16, 2020

ERROR: Service 'kafka' failed to build: #565

ERROR: Service 'kafka' failed to build: #565

Comments

ckvtvm-arch commented Jan 22, 2020

bellhop commented Jan 22, 2020

ckvtvm-arch commented Jan 22, 2020

ckvtvm-arch commented Jan 22, 2020

shellcheck disable=SC1091

Test to see if the suggested mirror has this version, currently pre 2.1.1 versions

do not appear to be actively mirrored. This may also be useful if closer.cgi is down.

OneCricketeer commented Jan 22, 2020

dgecawich commented Jan 28, 2020 • edited Loading

OneCricketeer commented Jan 28, 2020 • edited Loading

ckvtvm-arch commented Jan 28, 2020 via email

OneCricketeer commented Jan 28, 2020

OneCricketeer commented Jan 28, 2020 • edited Loading

dgecawich commented Jan 28, 2020 • edited Loading

ckvtvm-arch commented Jan 28, 2020 via email

OneCricketeer commented Jan 28, 2020

OneCricketeer commented Jan 28, 2020

ckvtvm-arch commented Jan 28, 2020 via email

OneCricketeer commented Jan 28, 2020

dgecawich commented Jan 28, 2020

OneCricketeer commented Jan 28, 2020 • edited Loading

dgecawich commented Jan 28, 2020

ckvtvm-arch commented Jan 28, 2020

ckvtvm-arch commented Jan 28, 2020

OneCricketeer commented Jan 28, 2020

OneCricketeer commented Jan 28, 2020

ckvtvm-arch commented Jan 28, 2020

ckvtvm-arch commented Jan 28, 2020

ckvtvm-arch commented Jan 28, 2020

ccarv commented Jan 30, 2020

OneCricketeer commented Jan 30, 2020

ccarv commented Jan 30, 2020

costa92 commented Mar 3, 2020

OneCricketeer commented Mar 3, 2020

ckvtvm-arch commented Mar 3, 2020 via email

costa92 commented Mar 5, 2020

OneCricketeer commented Mar 5, 2020

ckvtvm-arch commented May 4, 2020

OneCricketeer commented May 4, 2020

ckvtvm-arch commented May 4, 2020 via email

dgecawich commented Jan 28, 2020 •

edited

Loading

OneCricketeer commented Jan 28, 2020 •

edited

Loading

OneCricketeer commented Jan 28, 2020 •

edited

Loading

dgecawich commented Jan 28, 2020 •

edited

Loading

OneCricketeer commented Jan 28, 2020 •

edited

Loading