Spark docker cluster issue #1044
Unanswered
CHARANKANTHREDDY
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We installed spark with 1 master & 2 worker containers in docker but if we give spark submit command it is running only in the master we can't able to see in the UI port with 8080 spark can anyone have idea about this issue
for the first time we are getting bellow logs in worker
24/05/07 04:00:53 INFO Utils: Successfully started service 'sparkWorker' on port 7000.
24/05/07 04:00:53 INFO Worker: Worker decommissioning not enabled.
24/05/07 04:00:53 INFO Worker: Starting Spark worker ...:7000 with 1 cores, 1024.0 MiB RAM
24/05/07 04:00:53 INFO Worker: Running Spark version 3.4.2
24/05/07 04:00:53 INFO Worker: Spark home: /opt/spark
24/05/07 04:00:54 INFO ResourceUtils: ==============================================================
24/05/07 04:00:54 INFO ResourceUtils: No custom resources configured for spark.worker.
24/05/07 04:00:54 INFO ResourceUtils: ==============================================================
24/05/07 04:00:54 INFO JettyUtils: Start Jetty spark-worker-b:8080 for WorkerUI
24/05/07 04:00:54 INFO Utils: Successfully started service 'WorkerUI' on port 8080.
24/05/07 04:00:55 INFO WorkerWebUI: Bound WorkerWebUI to spark-worker-b, and started at http://5adf71920965:8080
24/05/07 04:00:55 INFO Worker: Connecting to master spark-master:7077...
24/05/07 04:00:55 WARN Worker: Failed to connect to master spark-master:7077
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:110)
at org.apache.spark.deploy.worker.Worker$$anon$1.run(Worker.scala:313)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Failed to connect to spark-master/...:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:294)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:226)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
... 4 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: spark-master/172.24.0.6:7077
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
24/05/07 04:01:10 INFO Worker: Retrying connection to master (attempt # 1)
24/05/07 04:01:10 INFO Worker: Connecting to master spark-master:7077...
24/05/07 04:01:10 INFO TransportClientFactory: Successfully created connection to spark-master/172.24.0.6:7077 after 3 ms (0 ms spent in bootstraps)
24/05/07 04:01:10 INFO Worker: Successfully registered with master spark://:.*.*7077
I replaced actual port with *
Beta Was this translation helpful? Give feedback.
All reactions