Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] pulsar cluster create error: BookKeeper metadata doesn't exist in zookeeper #8751

Open
JashBook opened this issue Jan 6, 2025 · 0 comments
Open

[BUG] pulsar cluster create error: BookKeeper metadata doesn't exist in zookeeper #8751

JashBook opened this issue Jan 6, 2025 · 0 comments
Assignees
@JashBook
Labels
kind/bug Something isn't working
Milestone
Release 1.0.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Jan 6, 2025

Describe the bug

kbcli version
Kubernetes: v1.28.3-vke.17
KubeBlocks: 1.0.0-beta.21
kbcli: 1.0.0-beta.8

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: pulsar-rrxops
  namespace: default
  labels:
    app.kubernetes.io/instance: pulsar-rrxops
  annotations:
    resource.kubeblocks.io/ignore-constraint: "true"
    kubeblocks.io/enabled-pod-ordinal-svc: broker
    "kubeblocks.io/extra-env": '{"KB_PULSAR_BROKER_NODEPORT": "true"}'
spec:
  clusterDef: pulsar
  topology: pulsar-enhanced-cluster
  terminationPolicy: WipeOut
  services:
    - name: broker-bootstrap
      serviceName: broker-bootstrap
      componentSelector: broker
      spec:
        type: NodePort
        ports:
          - name: pulsar
            port: 6650
            targetPort: 6650
          - name: http
            port: 80
            targetPort: 8080
          - name: kafka-client
            port: 9092
            targetPort: 9092
    - name: zookeeper
      serviceName: zookeeper
      componentSelector: zookeeper
      spec:
        type: ClusterIP
        ports:
          - name: client
            port: 2181
            targetPort: 2181
  componentSpecs:
    - name: proxy
      serviceVersion: 3.0.2
      replicas: 1
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies-recovery
      serviceVersion: 3.0.2
      replicas: 1
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: broker
      serviceVersion: 3.0.2
      replicas: 3
      env:
        - name: KB_PULSAR_BROKER_NODEPORT
          value: "true"
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies
      serviceVersion: 3.0.2 
      replicas: 3
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: ledgers
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: journal
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: zookeeper
      serviceVersion: 3.0.2
      replicas: 3
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. Click on '....'
  2. Scroll down to '....'
  3. See error
➜  ~ kubectl get cluster pulsar-rrxops   
NAME            CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
pulsar-rrxops   pulsar               WipeOut              Creating   5m45s
➜  ~ kubectl get pod -l app.kubernetes.io/instance=pulsar-rrxops 
NAME                               READY   STATUS     RESTARTS   AGE
pulsar-rrxops-bookies-recovery-0   0/1     Init:0/1   0          5m50s
pulsar-rrxops-zookeeper-0          1/1     Running    0          5m50s
pulsar-rrxops-zookeeper-1          1/1     Running    0          5m50s
pulsar-rrxops-zookeeper-2          1/1     Running    0          5m50s

logs pod

kubectl logs pulsar-rrxops-bookies-recovery-0 check-bookies --tail 100
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:383) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.apply(InstanceIdCommand.java:49) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	... 4 more
[0.936s][info ][gc,heap,exit] Heap
[0.936s][info ][gc,heap,exit]  garbage-first heap   total 217088K, used 81890K [0x000000066f000000, 0x0000000800000000)
[0.936s][info ][gc,heap,exit]   region size 4096K, 20 young (81920K), 0 survivors (0K)
[0.936s][info ][gc,heap,exit]  Metaspace       used 15788K, committed 16064K, reserved 1114112K
[0.936s][info ][gc,heap,exit]   class space    used 1869K, committed 1984K, reserved 1048576K
JAVA_HOME not set, using java from PATH. (/usr/bin/java)
[0.003s][trace][gc,heap]   Maximum heap size 6726850560
[0.003s][trace][gc,heap]   Initial heap size 210214080
[0.003s][trace][gc,heap]   Minimum heap size 6815736
[0.004s][debug][gc,heap] Minimum heap 8388608  Initial heap 213909504  Maximum heap 6727663616
[0.004s][info ][gc     ] Using G1
[0.004s][info ][gc,init] Version: 17.0.7+7-Debian-1deb11u1 (release)
[0.004s][info ][gc,init] CPUs: 4 total, 4 available
[0.004s][info ][gc,init] Memory: 12830M
[0.004s][info ][gc,init] Large Page Support: Disabled
[0.004s][info ][gc,init] NUMA Support: Disabled
[0.004s][info ][gc,init] Compressed Oops: Enabled (Zero based)
[0.004s][info ][gc,init] Heap Region Size: 4M
[0.004s][info ][gc,init] Heap Min Capacity: 8M
[0.004s][info ][gc,init] Heap Initial Capacity: 204M
[0.004s][info ][gc,init] Heap Max Capacity: 6416M
[0.004s][info ][gc,init] Pre-touch: Disabled
[0.004s][info ][gc,init] Parallel Workers: 4
[0.004s][info ][gc,init] Concurrent Workers: 4
[0.004s][info ][gc,init] Concurrent Refinement Workers: 4
[0.004s][info ][gc,init] Periodic GC: Disabled
[0.005s][info ][gc,metaspace] CDS archive(s) mapped at: [0x0000000800000000-0x0000000800be2000-0x0000000800be2000), size 12460032, SharedBaseAddress: 0x0000000800000000, ArchiveRelocationMode: 0.
[0.005s][info ][gc,metaspace] Compressed class space mapped at: 0x0000000801000000-0x0000000841000000, reserved size: 1073741824
[0.005s][info ][gc,metaspace] Narrow klass base: 0x0000000800000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[0.364s][info ][safepoint   ] Safepoint "ICBufferFull", Time since last: 356941270 ns, Reaching safepoint: 137605 ns, At safepoint: 2043 ns, Total: 139648 ns
[0.608s][info ][safepoint   ] Safepoint "ICBufferFull", Time since last: 243123576 ns, Reaching safepoint: 121421 ns, At safepoint: 3024 ns, Total: 124445 ns
2025-01-06T09:10:30,954+0000 [main] INFO  org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized
2025-01-06T09:10:30,963+0000 [main] INFO  org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://pulsar-rrxops-zookeeper.default.svc:2181/ledgers : zkServers = pulsar-rrxops-zookeeper.default.svc:2181, ledgersRootPath = /ledgers.
2025-01-06T09:10:30,966+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
2025-01-06T09:10:30,966+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=pulsar-rrxops-bookies-recovery-0.pulsar-rrxops-bookies-recovery-headless.default.svc.cluster.local
2025-01-06T09:10:30,966+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.7
2025-01-06T09:10:30,966+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian
2025-01-06T09:10:30,966+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64
2025-01-06T09:10:30,967+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/pulsar/conf:::/pulsar/lib/com.auth0-java-jwt-4.3.0.jar:/pulsar/lib/com.auth0-jwks-rsa-0.22.0.jar:/pulsar/lib/com.beust-jcommander-1.82.jar:/pulsar/lib/com.carrotsearch-hppc-0.9.1.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-core-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jdk8-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.datatype-jackson-datatype-jsr310-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.14.2.jar:/pulsar/lib/com.fasterxml.jackson.module-jackson-module-parameter-names-2.14.2.jar:/pulsar/lib/com.github.ben-manes.caffeine-caffeine-2.9.1.jar:/pulsar/lib/com.github.oshi-oshi-core-java11-6.4.0.jar:/pulsar/lib/com.github.seancfoley-ipaddress-5.3.3.jar:/pulsar/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/pulsar/lib/com.google.api.grpc-proto-google-common-protos-2.9.0.jar:/pulsar/lib/com.google.auth-google-auth-library-credentials-1.4.0.jar:/pulsar/lib/com.google.auth-google-auth-library-oauth2-http-1.4.0.jar:/pulsar/lib/com.google.auto.value-auto-value-annotations-1.9.jar:/pulsar/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/pulsar/lib/com.google.code.gson-gson-2.8.9.jar:/pulsar/lib/com.google.errorprone-error_prone_annotations-2.5.1.jar:/pulsar/lib/com.google.guava-failureaccess-1.0.1.jar:/pulsar/lib/com.google.guava-guava-32.1.1-jre.jar:/pulsar/lib/com.google.guava-listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/pulsar/lib/com.google.http-client-google-http-client-1.41.0.jar:/pulsar/lib/com.google.http-client-google-http-client-gson-1.41.0.jar:/pulsar/lib/com.google.j2objc-j2objc-annotations-1.3.jar:/pulsar/lib/com.google.protobuf-protobuf-java-3.19.6.jar:/pulsar/lib/com.google.protobuf-protobuf-java-util-3.19.6.jar:/pulsar/lib/com.google.re2j-re2j-1.6.jar:/pulsar/lib/com.rabbitmq-amqp-client-5.5.3.jar:/pulsar/lib/com.squareup.okhttp3-logging-interceptor-4.9.3.jar:/pulsar/lib/com.squareup.okhttp3-okhttp-4.9.3.jar:/pulsar/lib/com.squareup.okio-okio-3.4.0.jar:/pulsar/lib/com.squareup.okio-okio-jvm-3.4.0.jar:/pulsar/lib/com.sun.activation-javax.activation-1.2.0.jar:/pulsar/lib/com.typesafe.netty-netty-reactive-streams-2.0.6.jar:/pulsar/lib/com.yahoo.datasketches-memory-0.8.3.jar:/pulsar/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/pulsar/lib/commons-cli-commons-cli-1.5.0.jar:/pulsar/lib/commons-codec-commons-codec-1.15.jar:/pulsar/lib/commons-configuration-commons-configuration-1.10.jar:/pulsar/lib/commons-io-commons-io-2.8.0.jar:/pulsar/lib/commons-lang-commons-lang-2.6.jar:/pulsar/lib/commons-logging-commons-logging-1.1.1.jar:/pulsar/lib/io.airlift-aircompressor-0.20.jar:/pulsar/lib/io.dropwizard.metrics-metrics-core-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-graphite-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar:/pulsar/lib/io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar:/pulsar/lib/io.etcd-jetcd-api-0.7.5.jar:/pulsar/lib/io.etcd-jetcd-common-0.7.5.jar:/pulsar/lib/io.etcd-jetcd-core-0.7.5.jar:/pulsar/lib/io.etcd-jetcd-grpc-0.7.5.jar:/pulsar/lib/io.grpc-grpc-all-1.55.3.jar:/pulsar/lib/io.grpc-grpc-alts-1.55.3.jar:/pulsar/lib/io.grpc-grpc-api-1.55.3.jar:/pulsar/lib/io.grpc-grpc-auth-1.55.3.jar:/pulsar/lib/io.grpc-grpc-context-1.55.3.jar:/pulsar/lib/io.grpc-grpc-core-1.55.3.jar:/pulsar/lib/io.grpc-grpc-grpclb-1.55.3.jar:/pulsar/lib/io.grpc-grpc-netty-1.55.3.jar:/pulsar/lib/io.grpc-grpc-netty-shaded-1.55.3.jar:/pulsar/lib/io.grpc-grpc-protobuf-1.55.3.jar:/pulsar/lib/io.grpc-grpc-protobuf-lite-1.55.3.jar:/pulsar/lib/io.grpc-grpc-rls-1.55.3.jar:/pulsar/lib/io.grpc-grpc-services-1.55.3.jar:/pulsar/lib/io.grpc-grpc-servlet-1.55.3.jar:/pulsar/lib/io.grpc-grpc-servlet-jakarta-1.55.3.jar:/pulsar/lib/io.grpc-grpc-stub-1.55.3.jar:/pulsar/lib/io.grpc-grpc-xds-1.55.3.jar:/pulsar/lib/io.gsonfire-gson-fire-1.8.5.jar:/pulsar/lib/io.jsonwebtoken-jjwt-api-0.11.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-impl-0.11.1.jar:/pulsar/lib/io.jsonwebtoken-jjwt-jackson-0.11.1.jar:/pulsar/lib/io.kubernetes-client-java-18.0.0.jar:/pulsar/lib/io.kubernetes-client-java-api-18.0.0.jar:/pulsar/lib/io.kubernetes-client-java-proto-18.0.0.jar:/pulsar/lib/io.netty-netty-buffer-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-dns-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-haproxy-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-http-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-http2-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-codec-socks-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-common-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-handler-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-handler-proxy-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-classes-macos-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/pulsar/lib/io.netty-netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar:/pulsar/lib/io.netty-netty-tcnative-boringssl-static-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-tcnative-classes-2.0.61.Final.jar:/pulsar/lib/io.netty-netty-transport-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-transport-classes-epoll-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-epoll-4.1.100.Final.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final-linux-x86_64.jar:/pulsar/lib/io.netty-netty-transport-native-unix-common-4.1.100.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-classes-io_uring-0.0.21.Final.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-aarch_64.jar:/pulsar/lib/io.netty.incubator-netty-incubator-transport-native-io_uring-0.0.21.Final-linux-x86_64.jar:/pulsar/lib/io.opencensus-opencensus-api-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-contrib-http-util-0.28.0.jar:/pulsar/lib/io.opencensus-opencensus-proto-0.2.0.jar:/pulsar/lib/io.perfmark-perfmark-api-0.26.0.jar:/pulsar/lib/io.prometheus-simpleclient-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_caffeine-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_hotspot-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_httpserver-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_jetty-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_log4j2-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_servlet_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_common-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel-0.16.0.jar:/pulsar/lib/io.prometheus-simpleclient_tracer_otel_agent-0.16.0.jar:/pulsar/lib/io.prometheus.jmx-collector-0.16.1.jar:/pulsar/lib/io.reactivex.rxjava3-rxjava-3.0.1.jar:/pulsar/lib/io.swagger-swagger-annotations-1.6.10.jar:/pulsar/lib/io.swagger-swagger-core-1.6.10.jar:/pulsar/lib/io.swagger-swagger-models-1.6.10.jar:/pulsar/lib/io.vertx-vertx-auth-common-4.3.8.jar:/pulsar/lib/io.vertx-vertx-bridge-common-4.3.8.jar:/pulsar/lib/io.vertx-vertx-core-4.3.8.jar:/pulsar/lib/io.vertx-vertx-grpc-4.3.5.jar:/pulsar/lib/io.vertx-vertx-web-4.3.8.jar:/pulsar/lib/io.vertx-vertx-web-common-4.3.8.jar:/pulsar/lib/jakarta.activation-jakarta.activation-api-1.2.2.jar:/pulsar/lib/jakarta.annotation-jakarta.annotation-api-1.3.5.jar:/pulsar/lib/jakarta.validation-jakarta.validation-api-2.0.2.jar:/pulsar/lib/jakarta.ws.rs-jakarta.ws.rs-api-2.1.6.jar:/pulsar/lib/jakarta.xml.bind-jakarta.xml.bind-api-2.3.3.jar:/pulsar/lib/javax.annotation-javax.annotation-api-1.3.2.jar:/pulsar/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/pulsar/lib/javax.validation-validation-api-1.1.0.Final.jar:/pulsar/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/pulsar/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/pulsar/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/pulsar/lib/jline-jline-2.14.6.jar:/pulsar/lib/net.java.dev.jna-jna-jpms-5.12.1.jar:/pulsar/lib/net.java.dev.jna-jna-platform-jpms-5.12.1.jar:/pulsar/lib/net.jcip-jcip-annotations-1.0.jar:/pulsar/lib/net.jodah-failsafe-2.4.4.jar:/pulsar/lib/net.jodah-typetools-0.5.0.jar:/pulsar/lib/org.apache.avro-avro-1.11.3.jar:/pulsar/lib/org.apache.avro-avro-protobuf-1.11.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-proto-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-slogger-slf4j-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-circe-checksum-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-cpu-affinity-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-native-io-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-statelib-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-common-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-proto-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper-stream-storage-service-impl-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.http-http-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.http-vertx-http-server-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.16.3.jar:/pulsar/lib/org.apache.commons-commons-collections4-4.4.jar:/pulsar/lib/org.apache.commons-commons-compress-1.21.jar:/pulsar/lib/org.apache.commons-commons-lang3-3.11.jar:/pulsar/lib/org.apache.commons-commons-text-1.10.0.jar:/pulsar/lib/org.apache.curator-curator-client-5.1.0.jar:/pulsar/lib/org.apache.curator-curator-framework-5.1.0.jar:/pulsar/lib/org.apache.curator-curator-recipes-5.1.0.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-common-4.16.3.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3-tests.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-core-4.16.3.jar:/pulsar/lib/org.apache.distributedlog-distributedlog-protocol-4.16.3.jar:/pulsar/lib/org.apache.httpcomponents-httpclient-4.5.13.jar:/pulsar/lib/org.apache.httpcomponents-httpcore-4.4.15.jar:/pulsar/lib/org.apache.logging.log4j-log4j-api-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-core-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar:/pulsar/lib/org.apache.logging.log4j-log4j-web-2.18.0.jar:/pulsar/lib/org.apache.pulsar-bouncy-castle-bc-3.0.2-pkg.jar:/pulsar/lib/org.apache.pulsar-managed-ledger-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-oidc-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-auth-sasl-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-broker-common-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-admin-original-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-auth-sasl-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-messagecrypto-bc-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-original-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-client-tools-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-common-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-config-validation-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-api-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-instance-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-local-runner-original-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-proto-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-runtime-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-secrets-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-utils-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-functions-worker-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-io-core-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-metadata-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-bookkeeper-storage-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-core-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-package-filesystem-storage-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-proxy-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-testclient-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-common-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-transaction-coordinator-3.0.2.jar:/pulsar/lib/org.apache.pulsar-pulsar-websocket-3.0.2.jar:/pulsar/lib/org.apache.pulsar-testmocks-3.0.2.jar:/pulsar/lib/org.apache.thrift-libthrift-0.14.2.jar:/pulsar/lib/org.apache.yetus-audience-annotations-0.12.0.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-3.8.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-jute-3.8.3.jar:/pulsar/lib/org.apache.zookeeper-zookeeper-prometheus-metrics-3.8.3.jar:/pulsar/lib/org.asynchttpclient-async-http-client-2.12.1.jar:/pulsar/lib/org.asynchttpclient-async-http-client-netty-utils-2.12.1.jar:/pulsar/lib/org.bitbucket.b_c-jose4j-0.9.3.jar:/pulsar/lib/org.bouncycastle-bcpkix-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcprov-ext-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcprov-jdk15on-1.69.jar:/pulsar/lib/org.bouncycastle-bcutil-jdk15on-1.69.jar:/pulsar/lib/org.checkerframework-checker-qual-3.33.0.jar:/pulsar/lib/org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-conscrypt-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-alpn-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-client-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-continuation-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-http-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-io-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-proxy-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-security-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-servlets-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty-jetty-util-ajax-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-api-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-client-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-common-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-server-9.4.53.v20231009.jar:/pulsar/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.53.v20231009.jar:/pulsar/lib/org.glassfish.hk2-hk2-api-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-locator-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-hk2-utils-2.6.1.jar:/pulsar/lib/org.glassfish.hk2-osgi-resource-locator-1.0.3.jar:/pulsar/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.6.1.jar:/pulsar/lib/org.glassfish.hk2.external-jakarta.inject-2.6.1.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.34.jar:/pulsar/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-client-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-common-2.34.jar:/pulsar/lib/org.glassfish.jersey.core-jersey-server-2.34.jar:/pulsar/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.34.jar:/pulsar/lib/org.glassfish.jersey.inject-jersey-hk2-2.34.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.34.jar:/pulsar/lib/org.glassfish.jersey.media-jersey-media-multipart-2.34.jar:/pulsar/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/pulsar/lib/org.javassist-javassist-3.25.0-GA.jar:/pulsar/lib/org.jctools-jctools-core-2.1.2.jar:/pulsar/lib/org.jetbrains-annotations-13.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-common-1.6.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk7-1.6.0.jar:/pulsar/lib/org.jetbrains.kotlin-kotlin-stdlib-jdk8-1.6.0.jar:/pulsar/lib/org.jline-jline-3.21.0.jar:/pulsar/lib/org.jvnet.mimepull-mimepull-1.9.13.jar:/pulsar/lib/org.reactivestreams-reactive-streams-1.0.3.jar:/pulsar/lib/org.roaringbitmap-RoaringBitmap-0.9.44.jar:/pulsar/lib/org.rocksdb-rocksdbjni-7.9.2.jar:/pulsar/lib/org.slf4j-jcl-over-slf4j-1.7.32.jar:/pulsar/lib/org.slf4j-slf4j-api-1.7.32.jar:/pulsar/lib/org.xerial.snappy-snappy-java-1.1.10.5.jar:/pulsar/lib/org.yaml-snakeyaml-2.0.jar:
2025-01-06T09:10:30,967+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.4.250-4.ve1.x86_64
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=138MB
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=6416MB
2025-01-06T09:10:30,968+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=212MB
2025-01-06T09:10:30,971+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-rrxops-zookeeper.default.svc:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@7e985ce9
2025-01-06T09:10:30,974+0000 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2025-01-06T09:10:30,979+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
2025-01-06T09:10:30,985+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2025-01-06T09:10:30,993+0000 [main-SendThread(pulsar-rrxops-zookeeper.default.svc:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-rrxops-zookeeper.default.svc/172.30.44.59:2181.
2025-01-06T09:10:30,993+0000 [main-SendThread(pulsar-rrxops-zookeeper.default.svc:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2025-01-06T09:10:30,998+0000 [main-SendThread(pulsar-rrxops-zookeeper.default.svc:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.16.0.42:36466, server: pulsar-rrxops-zookeeper.default.svc/172.30.44.59:2181
2025-01-06T09:10:31,004+0000 [main-SendThread(pulsar-rrxops-zookeeper.default.svc:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-rrxops-zookeeper.default.svc/172.30.44.59:2181, session id = 0x2000027f2b00021, negotiated timeout = 30000
2025-01-06T09:10:31,006+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2025-01-06T09:10:31,026+0000 [main] ERROR org.apache.bookkeeper.discover.ZKRegistrationManager - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat
2025-01-06T09:10:31,130+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x2000027f2b00021 closed
2025-01-06T09:10:31,130+0000 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x2000027f2b00021
2025-01-06T09:10:31,130+0000 [main] ERROR org.apache.bookkeeper.bookie.BookieShell - Got an exception
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.apply(InstanceIdCommand.java:61) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.bookie.BookieShell$WhatIsInstanceId.runCmd(BookieShell.java:1498) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:248) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2349) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2446) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
Caused by: java.util.concurrent.ExecutionException: org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:363) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:383) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.apply(InstanceIdCommand.java:49) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	... 4 more
Caused by: org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
	at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:429) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.lambda$apply$0(InstanceIdCommand.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.lambda$runFunctionWithRegistrationManager$0(MetadataDrivers.java:386) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:358) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:383) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.apply(InstanceIdCommand.java:49) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	... 4 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
	at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:419) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.lambda$apply$0(InstanceIdCommand.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.lambda$runFunctionWithRegistrationManager$0(MetadataDrivers.java:386) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:358) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:383) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	at org.apache.bookkeeper.tools.cli.commands.bookies.InstanceIdCommand.apply(InstanceIdCommand.java:49) ~[org.apache.bookkeeper-bookkeeper-server-4.16.3.jar:4.16.3]
	... 4 more
[0.904s][info ][gc,heap,exit] Heap
[0.904s][info ][gc,heap,exit]  garbage-first heap   total 217088K, used 85984K [0x000000066f000000, 0x0000000800000000)
[0.904s][info ][gc,heap,exit]   region size 4096K, 20 young (81920K), 0 survivors (0K)
[0.904s][info ][gc,heap,exit]  Metaspace       used 15786K, committed 16064K, reserved 1114112K
[0.904s][info ][gc,heap,exit]   class space    used 1869K, committed 1984K, reserved 1048576K

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Jan 6, 2025
@JashBook JashBook added this to the Release 1.0.0 milestone Jan 6, 2025
@JashBook JashBook self-assigned this Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JashBook JashBook

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 1.0.0
Development

No branches or pull requests
1 participant
@JashBook