[KYUUBI #5380][UT] Create PySpark batch jobs tests for RESTful API #5498
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #5498 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 588 588
Lines 33466 33480 +14
Branches 4401 4405 +4
======================================
- Misses 33466 33480 +14 see 2 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiSuite.scala
Outdated
Show resolved
Hide resolved
kyuubi-server/src/test/scala/org/apache/kyuubi/BatchTestHelper.scala
Outdated
Show resolved
Hide resolved
| val requestObj = newBatchRequest( | ||
| batchType = PySparkJobPI.batchType, | ||
| resource = PySparkJobPI.resource.get, | ||
| className = null, |
There was a problem hiding this comment.
As mentioned in PR descriptions, it's a key point to set the className here to null. Is it applicable to all pyspark jobs ? If so , whether we could have a follow-up PR to skip sending className the pyspark batch type?
There was a problem hiding this comment.
If we use the RestAPI client, It need more effort to achieve this.
please refer to the dto:
https://github.com/apache/kyuubi/blob/ae29440453c35f4e541fb286effd6561d5234ba9/kyuubi-rest-client/src/main/java/org/apache/kyuubi/client/api/v1/dto/BatchRequest.java#L27C28-L27C28
Signed-off-by: weixi <[email protected]>
...bi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiPySparkSuite.scala
Outdated
Show resolved
Hide resolved
...bi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiPySparkSuite.scala
Outdated
Show resolved
Hide resolved
...bi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiPySparkSuite.scala
Outdated
Show resolved
Hide resolved
kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: weixi <[email protected]>
Signed-off-by: weixi <[email protected]>
...bi-server/src/test/scala/org/apache/kyuubi/server/rest/client/PySparkBatchRestApiSuite.scala
Outdated
Show resolved
Hide resolved
…ent/PySparkBatchRestApiSuite.scala
bowenliang123
approved these changes
Oct 24, 2023
|
@weixi62961 Thank you for your contribution and patience. |
bowenliang123
added a commit
that referenced
this pull request
Oct 25, 2023
### _Why are the changes needed?_ To close #5380. As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs. According the existing Spark Jar unit tests, two PySpark job unit test were added, they are all simple PI computing jobs from Spark examples. #### case1, "pyspark submit - basic batch rest client with existing resource file" It's almost same with the spark jar job test case, except the following two points: 1. param `batchType` should be set to `PYSPARK`, not `SPARK`. please refer to #3836 for detailed information. 2. For PySpark job，param `className` is useless， should be set to null #### case2, "pyspark submit - basic batch rest client with uploading resource file" Through the two test cases, simple PySpark jobs can be submitted normally. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5498 from weixi62961/unittest-batchapi-pyspark-simple. Closes #5380 b693efc [Bowen Liang] simplify sparkBatchTestResource 72a92b5 [Bowen Liang] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/PySparkBatchRestApiSuite.scala b2035a3 [weixi] remove no necessary wrapper object "PySparkJobPI" 27d12e8 [weixi] rename from BatchRestApiPySparkSuite to PySparkBatchRestApiSuite e680e60 [weixi] Create a dedicated batch API suite for PySpark jobs. dc8b6bf [weixi] add 2 test cases for pyspark batch job submit. Lead-authored-by: weixi <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Signed-off-by: liangbowen <[email protected]> (cherry picked from commit 5cff4fb) Signed-off-by: liangbowen <[email protected]>
|
Backported this PR to branch-1.8 (1.8.0) as well. No feature change is included. |
davidyuan1223
pushed a commit
to davidyuan1223/kyuubi
that referenced
this pull request
Oct 26, 2023
### _Why are the changes needed?_ To close apache#5380. As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs. According the existing Spark Jar unit tests, two PySpark job unit test were added, they are all simple PI computing jobs from Spark examples. #### case1, "pyspark submit - basic batch rest client with existing resource file" It's almost same with the spark jar job test case, except the following two points: 1. param `batchType` should be set to `PYSPARK`, not `SPARK`. please refer to apache#3836 for detailed information. 2. For PySpark job，param `className` is useless， should be set to null #### case2, "pyspark submit - basic batch rest client with uploading resource file" Through the two test cases, simple PySpark jobs can be submitted normally. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes apache#5498 from weixi62961/unittest-batchapi-pyspark-simple. Closes apache#5380 b693efc [Bowen Liang] simplify sparkBatchTestResource 72a92b5 [Bowen Liang] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/PySparkBatchRestApiSuite.scala b2035a3 [weixi] remove no necessary wrapper object "PySparkJobPI" 27d12e8 [weixi] rename from BatchRestApiPySparkSuite to PySparkBatchRestApiSuite e680e60 [weixi] Create a dedicated batch API suite for PySpark jobs. dc8b6bf [weixi] add 2 test cases for pyspark batch job submit. Lead-authored-by: weixi <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Signed-off-by: liangbowen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
To close #5380.
As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs.
According the existing Spark Jar unit tests, two PySpark job unit test were added, they are all simple PI computing jobs from Spark examples.
case1, "pyspark submit - basic batch rest client with existing resource file"
It's almost same with the spark jar job test case, except the following two points:
batchTypeshould be set toPYSPARK, notSPARK. please refer to [SPARK] Support pyspark batch job by restful api #3836 for detailed information.classNameis useless， should be set to nullcase2, "pyspark submit - basic batch rest client with uploading resource file"
Through the two test cases, simple PySpark jobs can be submitted normally.
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
No