Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Qualification tool: Disqualify Databricks Photon jobs at app level #804

Closed
nartal1 opened this issue Feb 23, 2024 · 2 comments · Fixed by #886
Closed

[FEA] Qualification tool: Disqualify Databricks Photon jobs at app level #804

nartal1 opened this issue Feb 23, 2024 · 2 comments · Fixed by #886
Assignees
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Feb 23, 2024

Is your feature request related to a problem? Please describe.
This issue to disqualify jobs if the eventlogs are from Databricks photon. Currently, eventlogs are read and the Photon Execs are marked as unsupported since the Exec names are different from regular CPU Execs. Ex: PhotonBroadcastExchange -> BroadcastExchange.
We need to mark the job as Not Applicable if the eventlogs are from Photon runs until we add support for Photon eventlogs.

Issue to add support for Databricks photon eventlogs - #251

@nartal1 nartal1 added feature request New feature or request ? - Needs Triage labels Feb 23, 2024
@cindyyuanjiang cindyyuanjiang self-assigned this Mar 13, 2024
@amahussein amahussein added the core_tools Scope the core module (scala) label Mar 20, 2024
@amahussein
Copy link
Collaborator

Had a offline discussion and key take-aways are:

  • We should skip the photon jobs for better performance. We don't want to spend additional time processing these event logs.
  • We could mark them as SKIPPED/FAILURE in rapids_4_spark_qualification_output_status.csv.

In details:

The suggested proposal of adding a new column is not going to be generic. It should be in the wrapper to make sure that the details/comments reflect the executive summary filters. Examples:

  • The “Top-candidates” view is part of the wrapper. The Top-Candidates has recommendations of its own and we won’t be able to use that in the new column.
  • Extending this in the future based on cost-savings or any other functionality won’t be feasible.

@amahussein
Copy link
Collaborator

@cindyyuanjiang there is another open issue #845 to disqualify streaming-jobs as well.
It will be nice to come up with an implementation that can be easily extended to resolve 845 as well.

@amahussein amahussein changed the title [FEA] Qualification tool: Disqualify Databricks Photon jobs at app level. [FEA] Qualification tool: Disqualify Databricks Photon jobs at app level Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
4 participants