[KYUUBI #6108] Display the CPU time consumed by the statement in the Spark Engine tab

[XorSum]

🔍 Description

Issue References 🔗

This pull request fixes #6108

Describe Your Solution 🔧

Follow #6112

Display the run time and CPU time of statements or sessions in the Spark UI.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 76.47059% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 61.14%. Comparing base (3bc28fd) to head (aad2bbb).
Report is 3 commits behind head on master.

Files	Patch %	Lines
...rc/main/scala/org/apache/spark/ui/EnginePage.scala	66.66%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6113      +/-   ##
============================================
+ Coverage     61.11%   61.14%   +0.03%     
  Complexity       23       23              
============================================
  Files           624      624              
  Lines         37245    37260      +15     
  Branches       5041     5046       +5     
============================================
+ Hits          22762    22784      +22     
+ Misses        12022    12015       -7     
  Partials       2461     2461              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wForget]

@XorSum This PR seems to have many duplicates with #6112. Which one should we use?

[XorSum]

@XorSum This PR seems to have many duplicates with #6112. Which one should we use?

Both.
#6112 collects the CPU time of operations, and #6113 displays the CPU time in Spark UI.
#6113 should rebase onto #6112.

[wForget]

@XorSum This PR seems to have many duplicates with #6112. Which one should we use?

Both. #6112 collects the CPU time of operations, and #6113 displays the CPU time in Spark UI. #6113 should rebase onto #6112.

Got it, you may need to rebase this change after #6112 is merged

[XorSum]

Got it, you may need to rebase this change after #6112 is merged

rebased

[wForget]

Should we show sessionRunTime instead of sessionCpuTime?

• sessionRunTime: The total running time of all tasks
• sessionCpuTime: The total cpu time occupied by all task threads (exclude io and other asynchronous operations?)

cc @pan3793 @yaooqinn @cxzl25

[cxzl25]

Should we show sessionRunTime instead of sessionCpuTime?

Maybe we can display both values, they are two dimensions of data.

[wForget]

Maybe we can display both values, they are two dimensions of data.

SGTM, @XorSum what do you think?

[XorSum]

Maybe we can display both values, they are two dimensions of data.

SGTM, @XorSum what do you think?

SGTM

[cxzl25]

Thank you for your contribution.
Tested locally, it looks good, but we need to update the run time of the unclosed session immediately.

We can just merge the two methods into one by the way.

org.apache.kyuubi.engine.spark.session.SparkSessionImpl#increaseRunTime
org.apache.kyuubi.engine.spark.session.SparkSessionImpl#increaseCpuTime

[XorSum]

@cxzl25 done.

[cxzl25]

Thanks, merged to master.