init_spark_on_k8s on remote node issues

[hkvision]

Scenario: I use a remote node to submit to the target k8s cluster. On the remote node, I use pip install in the conda environment for bigdl and related packages. In this case, the python environment on the driver (remote note) and executor (docker image) are different and thus there's bug in init_spark_on_k8s in our code.

Issues:

• python_env for PYTHONHOME, ld_path and preload_so should be the executor's python path https://github.com/intel-analytics/BigDL/blob/branch-2.0/python/dllib/src/bigdl/dllib/utils/spark.py#L324 Fixed in Fix init_spark_on_k8s #3589
• Users currently need to set the environment variable export PYSPARK_PYTHON=/usr/local/envs/pytf1/bin/python which is the python interpreter inside the docker image. support conda pack in init_spark_on_k8s #3646
• The jar paths for the executor extra classpath is the path on the driver, which is not correct and not available on the executor: https://github.com/intel-analytics/BigDL/blob/branch-2.0/python/dllib/src/bigdl/dllib/utils/spark.py#L348
  Should be ${BIGDL_HOME}/jars/* in the docker image. support conda pack in init_spark_on_k8s #3646
• spark.driver.host should be the ip of the remote node. Added the default value in Fix init_spark_on_k8s #3589 in case users may set this value wrong. May remove the spark.driver.host in our documentation since users don't need to specify it actually?
• Add a default value for spark.driver.port so that users don't need to set this as well in Fix init_spark_on_k8s #3589
• The NFS path should be also available and mounted to /bigdl2.0/data on the remote node. Spark requires driver to have access to the data as well.
• Add instructions for running k8s from a remote node in the documentation. Update k8s user guide for run from remote #3606
• Update the k8s user guide, remove driver host and port from the code and add a remark. Update k8s user guide for run from remote #3606
• RayOnSpark would fail when running k8s from remote #3605 This may be a little bit complicated and thus separate into a new issue.
• add init_orca_context(cluster_mode="k8s-cluster") & init_spark_on_k8s_cluster

[hkvision]

@jason-dai Any suggestion to set the executor classpath on the driver under the case that the driver and executor have different Python environment?
Can we hardcode the path in init_spark_on_k8s? Since in our image, we can fix the conda install path and bigdl path. We can hardcode PYSPARK_PYTHON and BIGDL_CLASSPATH? Also we may check whether the input container image contains bigdl-k8s as for other images not related to bigdl, we can't support.
Or any other preferred ways?
cc @glorysdj @Le-Zheng

[Le-Zheng]

@jason-dai Any suggestion to set the executor classpath on the driver under the case that the driver and executor have different Python environment? Can we hardcode the path in init_spark_on_k8s? Since in our image, we can fix the conda install path and bigdl path. We can hardcode PYSPARK_PYTHON and BIGDL_CLASSPATH? Also we may check whether the input container image contains bigdl-k8s as for other images not related to bigdl, we can't support. Or any other preferred ways? cc @glorysdj @Le-Zheng

Perhaps, we can pack conda python env into an archive file and ship that by "spark-submit --archives", which is similar to what we implement on yarn? @glorysdj
And we will not provide python env in the image.

[glorysdj]

@jason-dai Any suggestion to set the executor classpath on the driver under the case that the driver and executor have different Python environment? Can we hardcode the path in init_spark_on_k8s? Since in our image, we can fix the conda install path and bigdl path. We can hardcode PYSPARK_PYTHON and BIGDL_CLASSPATH? Also we may check whether the input container image contains bigdl-k8s as for other images not related to bigdl, we can't support. Or any other preferred ways? cc @glorysdj @Le-Zheng

Perhaps, we can pack conda python env into an archive file and ship that by "spark-submit --archives", which is similar to what we implement on yarn? @glorysdj And we will not provide python env in the image.

k8s can support this with archives option and uploading conda/pex envs.

[Le-Zheng]

@jason-dai Any suggestion to set the executor classpath on the driver under the case that the driver and executor have different Python environment? Can we hardcode the path in init_spark_on_k8s? Since in our image, we can fix the conda install path and bigdl path. We can hardcode PYSPARK_PYTHON and BIGDL_CLASSPATH? Also we may check whether the input container image contains bigdl-k8s as for other images not related to bigdl, we can't support. Or any other preferred ways? cc @glorysdj @Le-Zheng

Perhaps, we can pack conda python env into an archive file and ship that by "spark-submit --archives", which is similar to what we implement on yarn? @glorysdj And we will not provide python env in the image.

k8s can support this with archives option and uploading conda/pex envs.

Uploading conda envs on k8s cluster mode is tested successfully. Users can submit from remote without launching a k8s image client container.

1. user creates a conda python env. e.g, create env named "pynew" conda create -y -n pynew -c conda-forge conda-pack
2. pack this python env

conda activate pynew
conda pack -f -o pyspark_python.tar.gz

3. spark-submit with "--archives" and specify python stores for dirver and executor
   cluster mode:

--conf spark.pyspark.driver.python=./env/bin/python \
--conf spark.pyspark.python=./env/bin/python \ 
--archives local:///bigdl2.0/data/pyspark_python.tar.gz#env \  # this path shoud be what k8s pod can access

[hkvision]

@jason-dai Any suggestion to set the executor classpath on the driver under the case that the driver and executor have different Python environment? Can we hardcode the path in init_spark_on_k8s? Since in our image, we can fix the conda install path and bigdl path. We can hardcode PYSPARK_PYTHON and BIGDL_CLASSPATH? Also we may check whether the input container image contains bigdl-k8s as for other images not related to bigdl, we can't support. Or any other preferred ways? cc @glorysdj @Le-Zheng

Perhaps, we can pack conda python env into an archive file and ship that by "spark-submit --archives", which is similar to what we implement on yarn? @glorysdj And we will not provide python env in the image.

k8s can support this with archives option and uploading conda/pex envs.

Uploading conda envs on k8s cluster mode is tested successfully. Users can submit from remote without launching a k8s image client container.

1. user creates a conda python env. e.g, create env named "pynew" conda create -y -n pynew -c conda-forge conda-pack
2. pack this python env

conda activate pynew
conda pack -f -o pyspark_python.tar.gz

3. spark-submit with "--archives" and specify python stores for dirver and executor
   cluster mode:

--conf spark.pyspark.driver.python=./env/bin/python \
--conf spark.pyspark.python=./env/bin/python \ 
--archives local:///bigdl2.0/data/pyspark_python.tar.gz#env \  # this path shoud be what k8s pod can access

--archives local:///bigdl2.0/data/pyspark_python.tar.gz#env \ # this path shoud be what k8s pod can access

Do you mean each pod should have access to this path?
Or spark will automatically distribute the archive to each pod?

[Le-Zheng]

@jason-dai Any suggestion to set the executor classpath on the driver under the case that the driver and executor have different Python environment? Can we hardcode the path in init_spark_on_k8s? Since in our image, we can fix the conda install path and bigdl path. We can hardcode PYSPARK_PYTHON and BIGDL_CLASSPATH? Also we may check whether the input container image contains bigdl-k8s as for other images not related to bigdl, we can't support. Or any other preferred ways? cc @glorysdj @Le-Zheng

Perhaps, we can pack conda python env into an archive file and ship that by "spark-submit --archives", which is similar to what we implement on yarn? @glorysdj And we will not provide python env in the image.

k8s can support this with archives option and uploading conda/pex envs.

Uploading conda envs on k8s cluster mode is tested successfully. Users can submit from remote without launching a k8s image client container.

1. user creates a conda python env. e.g, create env named "pynew" conda create -y -n pynew -c conda-forge conda-pack
2. pack this python env

conda activate pynew
conda pack -f -o pyspark_python.tar.gz

3. spark-submit with "--archives" and specify python stores for dirver and executor
   cluster mode:

--conf spark.pyspark.driver.python=./env/bin/python \
--conf spark.pyspark.python=./env/bin/python \ 
--archives local:///bigdl2.0/data/pyspark_python.tar.gz#env \  # this path shoud be what k8s pod can access

--archives local:///bigdl2.0/data/pyspark_python.tar.gz#env \ # this path shoud be what k8s pod can access

Do you mean each pod should have access to this path? Or spark will automatically distribute the archive to each pod?

Spark will automatically distribute the archive to each pod.
For k8s client mode, the driver runs in the local physical host, and Spark can ship the local archive on the executor pod. The archive path does not have to be nfs. Spark submit command is:

--conf spark.pyspark.driver.python=python \
--conf spark.pyspark.python=./env/bin/python \ 
--archives local:///opt/work/pyspark_python.tar.gz#env

For k8s cluster mode, the driver runs in k8s pod. The archive path must be where k8s pod can access, such as nfs. Spark submit command is:

--conf spark.pyspark.driver.python=./env/bin/python \
--conf spark.pyspark.python=./env/bin/python \ 
--archives local:///bigdl2.0/data/pyspark_python.tar.gz#env

[hkvision]

great, this makes sense.