[FEATURE REQUEST] Quick Start: Spark via docker

[kevinjqliu]

Is your feature request related to a problem? Please describe.

Not a problem. Enhancements to the "Quick Start" guide.

Describe the solution you'd like

Include Spark docker container instead of cloning the Spark GitHub repo as described in the "Quick Start" guide

Describe alternatives you've considered

Possibly use PySpark or other engines (Trino, PyIceberg, etc)

Additional context

Modelled after the Spark and Iceberg Quickstart guide, which is defined in https://github.com/tabular-io/docker-spark-iceberg/tree/main

[collado-mike]

There is a jupyter notebook example using docker-compose at https://github.com/apache/polaris/blob/main/docker-compose-jupyter.yml . It's not in the quickstart guide, though.

[kevinjqliu]

@collado-mike wdyt about moving that to the getting-started/ folder as part of #42
alternatively, we can create a new example with just spark

[kevinjqliu]

Possibly include #192 as setup script

[flyrain]

Other than the docker option, ./regtest/run_spark_sql.sh is even a faster option to connect to Polaris with Spark. Here is the usage:

# Usage:
#   ./run_spark_sql.sh [S3-location AWS-IAM-role]
#
# Description:
#   - Without arguments: Runs against a catalog backed by the local filesystem.
#   - With two arguments: Runs against a catalog backed by AWS S3.
#       - [S3-location]  - The S3 path to use as the default base location for the catalog.
#       - [AWS-IAM-role] - The AWS IAM role for catalog to assume when accessing the S3 location.
#
# Examples:
#   - Run against local filesystem:
#     ./run_spark_sql.sh
#
#   - Run against AWS S3:
#     ./run_spark_sql.sh s3://my-bucket/path arn:aws:iam::123456789001:role/my-role

[kevinjqliu]

run_spark_sql.sh requires a running Polaris service, right? I want to include it as part of the "getting-started" for Spark.
One option is jupyter, the other is this script which spawns a spark sql shell

[flyrain]

run_spark_sql.sh requires a running Polaris service, right?

Yes. Feel free to add it in the doc, and thank you for doing this!