Running Spark Tasks on Flyte

[SandraGH5]

I have some trouble running Spark tasks on Flyte.
The simple parametric hello world works fine:

• full Flyte up and running on k8s cluster
• have the flyte-sparkoperator up and healthy

Steps I did:

• I have copied to "myflyteapp" the pyspark_pi workflow
• packaged it up

./docker_build_and_tag.sh
pyflyte --pkgs flyte.workflows package --image <registry/repo:version>
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v3

Then I triggered it from WebUI
Running it results in a failure:

• failed pod

No images specified, will use the default image                                                                                      
│ Traceback (most recent call last):                                                                                                   
│   File "/opt/venv/bin/pyflyte-execute", line 8, in <module>                                                                          
│     sys.exit(execute_task_cmd())
....
│   File "<frozen importlib._bootstrap_external>", line 843, in exec_module                                                            
│   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed                                                       
│   File "/root/flyte/workflows/pyspark_pi.py", line 75, in <module>                                                                   
│     from flytekitplugins.spark import Spark                                                                                          │
│ ModuleNotFoundError: No module named 'flytekitplugins'

In Web UI same error:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[kzaru3dlk9-n0-0] terminated with exit code (1). Reason [Error]. Message: 
t/f8f39d9abe810e4f997a5f59df9c91e0
No images specified, will use the default image
....
    from flytekitplugins.spark import Spark
ModuleNotFoundError: No module named 'flytekitplugins'

I have followed the steps in https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#step-2-environment-setup

Except for step 2. "Build Spark image"
I don't really understand what spark image I have to build and where to plug it in ??

Isn't the sparkoperator supposed to do this? Where/how is this image referenced ?
What am I missing from setting this up?

@fbunau

[SandraGH5]

I think your example code doesn't have flytekitplugins-spark. Please check the example: https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/requirements.in

requirements.in

-r ../../../common/requirements-common.in
pyspark

flytekitplugins-spark>=0.16.0

Also, if Java is installed, set JAVA_HOME env variable in the Dockerfile, using a docker file like this:
https://github.com/flyteorg/flytesnacks/blob/master/cookbook/integrations/kubernetes/k8s_spark/Dockerfile which has java installed

Dockerfile

FROM ubuntu:focal
LABEL org.opencontainers.image.source https://github.com/flyteorg/flytesnacks

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
ENV DEBIAN_FRONTEND=noninteractive

# Install Python3 and other basics
RUN apt-get update && apt-get install -y python3.8 python3.8-venv make build-essential libssl-dev python3-pip curl

# Install AWS CLI to run on AWS (for GCS install GSutil). This will be removed
# in future versions to make it completely portable
RUN pip3 install awscli

# Install gcloud for GCP
RUN apt-get install curl --assume-yes

RUN curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:/root/google-cloud-sdk/bin

ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

# Install Python dependencies
COPY k8s_spark/requirements.txt /root
RUN pip install -r /root/requirements.txt

RUN flytekit_install_spark3.sh
# Adding Tini support for the spark pods
RUN wget  https://github.com/krallin/tini/releases/download/v0.18.0/tini && \
    cp tini /sbin/tini && cp tini /usr/bin/tini && \
    chmod a+x /sbin/tini && chmod a+x /usr/bin/tini

# Setup Spark environment
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV SPARK_HOME /opt/spark
ENV SPARK_VERSION 3.0.1
ENV PYSPARK_PYTHON ${VENV}/bin/python3
ENV PYSPARK_DRIVER_PYTHON ${VENV}/bin/python3

# Copy the makefile targets to expose on the container. This makes it easier to register.
COPY in_container.mk /root/Makefile
COPY k8s_spark/sandbox.config /root

# Copy the actual code
COPY k8s_spark/ /root/k8s_spark

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

# Copy over the helper script that the SDK relies on
RUN cp ${VENV}/bin/flytekit_venv /usr/local/bin/
RUN chmod a+x /usr/local/bin/flytekit_venv

# For spark we want to use the default entrypoint which is part of the
# distribution, also enable the virtualenv for this image. 
# Note this relies on the VENV variable we've set in this image.
ENTRYPOINT ["/usr/local/bin/flytekit_venv", "/opt/entrypoint.sh"]

We also suggest you to try step 7 & 8 in the spark example here: https://docs.flyte.org/en/latest/deployment/plugin_setup/k8s/spark_operator.html#deployment-plugin-setup-k8s-spark-operator

That will validate your spark environment, and then follow the Flytesnacks example here: https://github.com/flyteorg/flytesnacks/tree/master/cookbook/integrations/kubernetes/k8s_spark

@evalsocket
@samhita-alla