Exited spark-submit process should not block batch submit queue

[pan3793]

🔍 Description

Issue References 🔗

While enabling batch implementation V2 with the following configurations

kyuubi.batch.impl.version=2
kyuubi.batch.submitter.enabled=true
kyuubi.batch.submitter.threads=48
spark.master=yarn
spark.submit.deployMode=cluster
spark.yarn.submit.waitAppCompletion=false

I found that the batch jobs will be blocked in the DB queue once a YARN queue has no resources, this brings an issue, the subsequential batch jobs that are going to be submitted to another YARN queue also be queued in DB, rather than YARN queue.

mysql> select state, engine_state, count(1) from metadata where state in ('INITIALIZED', 'PENDING', 'RUNNING') group by state, engine_state;
+-------------+--------------+----------+
| state       | engine_state | count(1) |
+-------------+--------------+----------+
| INITIALIZED | NULL         |      166 |
| PENDING     | NULL         |        1 |
| RUNNING     | PENDING      |      148 |
| RUNNING     | RUNNING      |      415 |
+-------------+--------------+----------+

Describe Your Solution 🔧

The submitter queue whose size is controlled by kyuubi.batch.submitter.threads is designed to address the spark-submit process concurrency issue, too many spark-submit processes may run out of the Kyuubi server's node CPU/memory resources and eventually crash the service. For Spark YARN cluster mode, if set spark.yarn.submit.waitAppCompletion=false, the local spark-submit process exits immediately once the Application goes ACCEPTED status, even no resource could be allocated for the AM container, we should not block such case in submitter queue.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Pass GA, and roll out into internal cluster.

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (3f993f4) 61.11% compared to head (05fcc75) 61.07%.
Report is 2 commits behind head on master.

Files	Patch %	Lines
.../org/apache/kyuubi/server/KyuubiBatchService.scala	0.00%	1 Missing and 1 partial ⚠️
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala	50.00%	1 Missing ⚠️
...g/apache/kyuubi/operation/BatchJobSubmission.scala	0.00%	1 Missing ⚠️
...org/apache/kyuubi/session/KyuubiBatchSession.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6028      +/-   ##
============================================
- Coverage     61.11%   61.07%   -0.04%     
  Complexity       23       23              
============================================
  Files           623      623              
  Lines         37136    37140       +4     
  Branches       5031     5031              
============================================
- Hits          22695    22685      -10     
- Misses        11997    12005       +8     
- Partials       2444     2450       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

Thanks, merged to master/1.8