Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Spark docs v2 #4670

Closed
wants to merge 8 commits into from
Closed

Update Spark docs v2 #4670

wants to merge 8 commits into from

Conversation

davidmirror-ops
Copy link
Contributor

@davidmirror-ops davidmirror-ops commented Jan 3, 2024

Tracking issue

Closes #4545

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: davidmirror-ops <[email protected]>
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. documentation Improvements or additions to documentation labels Jan 3, 2024
Copy link

codecov bot commented Jan 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a726c36) 58.11% compared to head (5949354) 58.15%.
Report is 15 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4670      +/-   ##
==========================================
+ Coverage   58.11%   58.15%   +0.03%     
==========================================
  Files         626      626              
  Lines       53815    53786      -29     
==========================================
+ Hits        31275    31277       +2     
+ Misses      20039    20001      -38     
- Partials     2501     2508       +7     
Flag Coverage Δ
unittests 58.15% <ø> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: davidmirror-ops <[email protected]>

Enable the Spark plugin on the demo cluster by adding the following block to ~/.flyte/sandbox/config.yaml:

.. code-block:: yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs to be indented. i think you haven't copied the whole config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just checked and there are the same three sections in the live docs: tasks, plugins and cluster_resources. Also checked indentation and it should be fine. It uses the same spacing as in a working deployment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i haven't tested the spark one 'cause it didn't work for me earlier. but i believe this is how it needs to be done. but we need to test the config.

Signed-off-by: davidmirror-ops <[email protected]>
Copy link
Contributor

@samhita-alla samhita-alla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left one comment (apologies, should have checked it before); if you've already added, please merge the PR.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation lgtm This PR has been approved by a maintainer size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Docs] Update Spark operator configuration docs
2 participants