Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark Jupyter getting started docker compose #295

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Spark Jupyter getting started docker compose #295

wants to merge 2 commits into from

Conversation

kevinjqliu
Copy link
Contributor

Description

This PR moves the docker-compose-jupyter.yml file (and the notebooks/ directory), formerly in the top-level directory, into the getting-started/spark/ folder.

The purpose is to unify the "getting started" guides into the same directory.

Fixes #110

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Documentation update
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

docker-compose -f getting-started/spark/docker-compose-jupyter.yml up

Checklist:

Please delete options that are not relevant.

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

@kevinjqliu
Copy link
Contributor Author

I want to make sure this is something we want to do before proceeding to add more to the PR

cc @collado-mike / @flyrain

@flyrain
Copy link
Contributor

flyrain commented Sep 16, 2024

Make sense to me. Thanks @kevinjqliu! Do we have any doc for its usage? We may add doc if not.

@kevinjqliu
Copy link
Contributor Author

@flyrain yep i'll have a README in here, similar to the trino one

@flyrain
Copy link
Contributor

flyrain commented Sep 16, 2024

Sounds good. We will need these doc to be in the Polaris doc site, like this https://polaris.apache.org/docs/overview/. I couldn't find Trino's doc there, this may involve doc publish and link. cc @jbonofre

@kevinjqliu
Copy link
Contributor Author

I see, this is the README for trino. I'll add a similar README for spark.

As a follow-up, we can change the Polaris doc to refer to these guides https://polaris.apache.org/docs/quickstart

@collado-mike
Copy link
Contributor

This looks good to me. We should change the name of the compose file to just docker-compose.yml so we don't have to specify the filename in the command line :)

@kevinjqliu
Copy link
Contributor Author

@collado-mike makes sense, will do.

I have a question on slack about unable to assume the role arn:aws:iam::631484165566:role/datalake-storage-integration-role in the notebook, do you mind taking a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbonofre jbonofre Awaiting requested review from jbonofre jbonofre will be requested when the pull request is marked ready for review jbonofre is a code owner

@ashvina ashvina Awaiting requested review from ashvina ashvina will be requested when the pull request is marked ready for review ashvina is a code owner

@RussellSpitzer RussellSpitzer Awaiting requested review from RussellSpitzer RussellSpitzer will be requested when the pull request is marked ready for review RussellSpitzer is a code owner

@snazy snazy Awaiting requested review from snazy snazy will be requested when the pull request is marked ready for review snazy is a code owner

@vvcephei vvcephei Awaiting requested review from vvcephei vvcephei will be requested when the pull request is marked ready for review vvcephei is a code owner

@takidau takidau Awaiting requested review from takidau takidau will be requested when the pull request is marked ready for review takidau is a code owner

@jackye1995 jackye1995 Awaiting requested review from jackye1995 jackye1995 will be requested when the pull request is marked ready for review jackye1995 is a code owner

@flyrain flyrain Awaiting requested review from flyrain flyrain will be requested when the pull request is marked ready for review flyrain is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE REQUEST] Quick Start: Spark via docker
3 participants
@kevinjqliu @flyrain @collado-mike