[KYUUBI #5196][FOLLOWUP] Extract spark core scala version lazily and …

…respect engine env

### _Why are the changes needed?_

Only extract the spark core scala version if `SPARK_SCALA_VERSION` env is empty, and respect engine env.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5434 from turboFei/lazy_scala_version.

Closes #5196

fdccef7 [fwang12] lazy extract spark core scala version

Authored-by: fwang12 <[email protected]>
Signed-off-by: fwang12 <[email protected]>

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala

@@ -108,9 +108,9 @@ class SparkProcessBuilder(
   }
 
   override protected val engineScalaBinaryVersion: String = {
-    val sparkCoreScalaVersion =
+    env.get("SPARK_SCALA_VERSION").filter(StringUtils.isNotBlank).getOrElse {
       extractSparkCoreScalaVersion(Paths.get(sparkHome, "jars").toFile.list())
-    StringUtils.defaultIfBlank(System.getenv("SPARK_SCALA_VERSION"), sparkCoreScalaVersion)
+    }
   }
 
   override protected lazy val engineHomeDirFilter: FileFilter = file => {