[FEA] Skip Databricks Photon jobs at app level in Qualification tool

[cindyyuanjiang]

Fixes #804

This PR implements the logic to skip Databricks Photon jobs at app level in Qualification tool. Each Photon event log will be marked at SKIPPED in rapids_4_spark_qualification_output_status.csv and removed from other generated files.

Changes

• Add class SkippedQualAppResult to represent skipped Qualification App creation and skippedCounter for console progress bar visualization
• Look for keyword Photon when processing SQL plans and short-circuit the process if there exists one
• Add case class FailureApp for handling different kinds of exceptions when creating Qualification App

Tests

spark_rapids_user_tools databricks-azure qualification --cpu_cluster my_cpu_cluster --eventlogs my_event_logs --tools_jar my_tools_jar

Before this PR:

rapids_4_spark_qualification_output.csv

App Name,App ID,Recommendation,Estimated GPU Speedup,Estimated GPU Duration,Estimated GPU Time Saved,SQL DF Duration,SQL Dataframe Task Duration,App Duration,GPU Opportunity,Executor CPU Time Percent,SQL Ids with Failures,Unsupported Read File Formats and Types,Unsupported Write Data Format,Complex Types,Nested Complex Types,Potential Problems,Longest SQL Duration,NONSQL Task Duration Plus Overhead,Unsupported Task Duration,Supported SQL DF Task Duration,Task Speedup Factor,App Duration Estimated,Unsupported Execs,Unsupported Expressions,Estimated Job Frequency (monthly),Cluster Tags
"Databricks Shell","app-20240220023733-0000","Not Recommended",1.0,3520729.91,22972.08,501797,1682938390,3543702,41203,41.59,"","JSON[timestamp:map:string:array:boolean:date:int:struct:bigint:dec:byte]","","struct;struct,size:bigint,modificationTime:bigint,dataChange:boolean,stats:string,tags:map,deletionVector:struct,baseRowId:bigint,defaultRowCommitVersion:bigint,clusteringProvider:string>;struct,size:bigint,tags:map,deletionVector:struct,baseRowId:bigint,defaultRowCommitVersion:bigint>;struct>,schemaString:string,partitionColumns:array,configuration:map,createdTime:bigint>;struct,writerFeatures:array>;struct,size:bigint,tags:map>;struct>;struct>;struct;struct,job:struct,notebook:struct,clusterId:string,readVersion:bigint,isolationLevel:string,isBlindAppend:boolean,operationMetrics:map,userMetadata:string,tags:map,engineInfo:string,txnId:string>","struct,size:bigint,modificationTime:bigint,dataChange:boolean,stats:string,tags:map,deletionVector:struct,baseRowId:bigint,defaultRowCommitVersion:bigint,clusteringProvider:string>;struct,size:bigint,tags:map,deletionVector:struct,baseRowId:bigint,defaultRowCommitVersion:bigint>;struct>,schemaString:string,partitionColumns:array,configuration:map,createdTime:bigint>;struct,writerFeatures:array>;struct,size:bigint,tags:map>;struct>;struct>;struct,job:struct,notebook:struct,clusterId:string,readVersion:bigint,isolationLevel:string,isBlindAppend:boolean,operationMetrics:map,userMetadata:string,tags:map,engineInfo:string,txnId:string>","UDF:NESTED COMPLEX TYPE",241288,516242735,1544550551,138387839,2.25,true,"SerializeFromObject;PhotonShuffleMapStage;PhotonGroupingAgg;PhotonShuffleExchangeSink;PhotonShuffleExchangeSource;PhotonProject;MapPartitions;PhotonRowToColumnar;Execute ClearCacheCommand$;Scan ExistingRDD;DeserializeToObject;PhotonFilter;LocalTableScan;Project;Scan json;PhotonResultStage;ObjectHashAggregate;PhotonScan parquet ;MapElements;PhotonParquetWriter;Execute OptimizeTableCommandEdge;PhotonWriteStage","UDF;from_json;filesizehistogramagg",30,"ClusterId -> 0220-023022-xvizld3;created_by -> at657h;DatabricksEnvironment -> workerenv-2876790984019916;ClusterName -> job-518534639051370-run-104068185572167-deDup_lteDist_cluster;BU_Name -> IQI-Standard-Compute-Pool;JobId -> 518534639051370;RunName -> DeviceAnalyzer Production;Creator -> [email protected];Vendor -> Databricks;mots_id -> 30277;contact_dl -> [email protected];env -> prod;BU_Mots_ID -> 26015"

rapids_4_spark_qualification_output_status.csv

Event Log,Status,Description
".../photon-eventlogs","SUCCESS","app-20240220023733-0000,Took 37812ms to process"

Console Log

| 24/03/26 14:04:08 INFO SuccessQualAppResult: File: .../photon-eventlogs, Message: Took 37812ms to process
| Qual Tool Progress 100% [=========================================================] (1 succeeded + 0 failed + 0 N/A) / 1
| 
| Qual Tool execution time: 37849ms
| 	process.success.count = 1
| 	process.failure.count = 0
| 	process.NA.count = 0
| 	execution.total.count = 1

After this PR:

rapids_4_spark_qualification_output.csv

App Name,App ID,Recommendation,Estimated GPU Speedup,Estimated GPU Duration,Estimated GPU Time Saved,SQL DF Duration,SQL Dataframe Task Duration,App Duration,GPU Opportunity,Executor CPU Time Percent,SQL Ids with Failures,Unsupported Read File Formats and Types,Unsupported Write Data Format,Complex Types,Nested Complex Types,Potential Problems,Longest SQL Duration,NONSQL Task Duration Plus Overhead,Unsupported Task Duration,Supported SQL DF Task Duration,Task Speedup Factor,App Duration Estimated,Unsupported Execs,Unsupported Expressions,Estimated Job Frequency (monthly)

rapids_4_spark_qualification_output_status.csv

Event Log,Status,Description
".../photon-eventlogs","SKIPPED","PhotonEventLogException: Encountered Databricks Photon event log: skipping this file!"

Console Log

| 24/03/26 14:08:35 WARN SkippedQualAppResult: File: .../photon-eventlogs, Message: PhotonEventLogException: Encountered Databricks Photon event log: skipping this file!
| Qual Tool Progress 100% [=============================================] (0 succeeded + 0 failed + 1 skipped + 0 N/A) / 1
| 
| Qual Tool execution time: 683ms
| 	process.success.count = 0
| 	process.failure.count = 0
| 	process.skipped.count = 1
| 	process.NA.count = 0
| 	execution.total.count = 1

[nartal1]

Thanks @cindyyuanjiang ! Just a couple of nits.

[nartal1]

Nit: Does Photon operators always begin with Photon keyword. If so can we use startsWith instead of contains here.

[cindyyuanjiang]

I am not certain, but from previous observations in: #449 (comment), it looks like they begin with "Photon". I will update this.

[nartal1]

Nit: unknow -> unknown

[cindyyuanjiang]

Thanks, updated!

[amahussein]

Thanks @cindyyuanjiang !
I am fine with merging this as is for now, given that we investigate moving the logic of detection during the eventlog-processing phase as described in my comment.

No need to file a followup as\the investigation can be part of the existing #845, which is pretty similar to this feature.

[amahussein]

I thought that we could detect the Photon in an earlier phase as we process the eventlogs.
For example, doSparkListenerSQLExecutionStart() could check for certain conditions to disqualify the entire app.
The pros of having the code in doSparkListenerSQLExecutionStart():

• For Qualification tool, the bottleneck of performance is actually in reading and processing the evntlogs. Therefore, if we fail during that stage, then we are saving significant amount of useless processing and computation.
• If we need to propagate that logic to Profiling, it becomes much easier as we need only to move the implementation to the EvenProcessorBase.doSparkListenerSQLExecutionStart instead of QualificationEventProcessor.doSparkListenerSQLExecutionStart. Qualification if we need