Add sparkRuntime property to capture runtime type in application_information #1414
+48
−8
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1413
This PR adds a new
sparkRuntime
property to capture the Spark Runtime type(SPARK, PHOTON, SPARK_RAPIDS)
and store this inapplication_information.csv
Changes
Profiling Enhancements:
sparkRuntime
property toAppInfoProfileResults
to capture the runtime environment and updated theoutputHeaders
andconvertToSeq
methods to include this new property. [1] [2]AppInformationViewTrait
to map the newsparkRuntime
property when creatingAppInfoProfileResults
instances.Runtime Handling:
SparkRuntime
enumeration to represent different Spark runtimes (SPARK, PHOTON, SPARK_RAPIDS).sparkRuntime
property toCacheablePropsHandler
and set its default value.setSparkRuntime
method inAppBase
to determine and set the runtime based on application properties.Testing:
ApplicationInfoSuite
to validate thesparkRuntime
property for different event logs.Output
File:
application_information.csv
SPARK Runtime:
SPARK_RAPIDS Runtime:
PHOTON Runtime:
cc: @leewyang