Failed to start pyspark kernel on SoS

[ambamanyloic]

Hi Bo

Can you confirm that you can effectively write data into Hive from the spark_python_conductor_cluster kernel on SoS ?
Even a simple "import pyspark"command results in a Permission Denied on the '/tmp/...' folder
Attaching a screenshot

[ambamanyloic]

[BoPeng]

What is a spark_python_conductor_cluster kernel? Could you send me a link to the kernel so that I can try it on my mac?

[ambamanyloic]

spark_python_conductor_cluster is an IBM spark kernel provided by IBM to help with resources, slots ... it does not change anything to the embedded spark cluster setup itself , I can send you the kernel json if you'd like . Basically it seems that the combination of sos and spark cluster mode are not working well together , the spark cluster mode can not write to /tmp when it is running under sos , but when I use spark cluster by itself it works ....can you just try to do a "spark" under your spark cluster on sos and confirms it works ?

[BoPeng]

We do not have a spark cluster here and cannot test. Just looking at the message we might be looking at

https://github.com/vatlab/sos-notebook/blob/bfdac0f92168913bc3fb142e1e2e375da6a3333d/src/sos_notebook/kernel.py#L1493

The temp file stuff is used to capture error message from the kernel (if its second attempt to start also failed), so the root problem is that sos failed to start the kernel. Let me see how I can get my hands on a spark environment and test.

[ambamanyloic]

Ok thanks , let me know , it looks like for spark everything has to go through local but you re right , the spark kernel does not even start for me