Implement FlintJob to handle all query types in warmpool mode #979
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
saranrajnk
requested review from
dai-chen,
mengweieric,
penghuo,
seankao-az,
anirudha,
kaituo,
YANG-DB,
noCharger,
LantaoJin and
ykmr1224
as code owners
saranrajnk
force-pushed
the
nexus-wp-feat
branch
from
6747ab9 to
59aa26b
Compare
spark-sql-application/src/main/scala/org/apache/spark/sql/FlintJob.scala
Show resolved
Hide resolved
spark-sql-application/src/main/scala/org/apache/spark/sql/FlintJob.scala
Show resolved
Hide resolved
ykmr1224
requested changes
Dec 16, 2024
spark-sql-application/src/main/scala/org/apache/spark/sql/FlintJob.scala
Outdated
Show resolved
Hide resolved
spark-sql-application/src/main/scala/org/apache/spark/sql/FlintJob.scala
Outdated
Show resolved
Hide resolved
noCharger
reviewed
Dec 17, 2024
Comment on lines
550
to
644
| def getSegmentName(sparkSession: SparkSession): String = { | ||
| val maxExecutorsCount = | ||
| sparkSession.conf.get(FlintSparkConf.MAX_EXECUTORS_COUNT.key, "unknown") | ||
| String.format("%se", maxExecutorsCount) | ||
| } |
There was a problem hiding this comment.
This segmentName is specific to warmpool logic; let us create abstractions on warmpool and record metrics via AOP.
spark-sql-application/src/main/scala/org/apache/spark/sql/FlintREPL.scala
Outdated
Show resolved
Hide resolved
spark-sql-application/src/main/scala/org/apache/spark/sql/JobOperator.scala
Show resolved
Hide resolved
spark-sql-application/src/main/scala/org/apache/spark/sql/FlintJob.scala
Outdated
Show resolved
Hide resolved
saranrajnk
force-pushed
the
nexus-wp-feat
branch
3 times, most recently
from
044aeea to
adef5b6
Compare
ykmr1224
reviewed
Jan 7, 2025
spark-sql-application/src/main/scala/org/apache/spark/sql/WarmpoolJob.scala
Show resolved
Hide resolved
spark-sql-application/src/main/scala/org/apache/spark/sql/WarmpoolJob.scala
Outdated
Show resolved
Hide resolved
noCharger
requested changes
Jan 10, 2025
spark-sql-application/src/main/scala/org/apache/spark/sql/WarmpoolJob.scala
Outdated
Show resolved
Hide resolved
| } | ||
| } | ||
|
|
||
| def queryLoop(commandContext: CommandContext): Unit = { |
There was a problem hiding this comment.
why do we need the concept of query loop for warm pool?
There was a problem hiding this comment.
Warmpool requires multiple iterations as well before running the actual query.
saranrajnk
force-pushed
the
nexus-wp-feat
branch
from
e195862 to
b19028b
Compare
Signed-off-by: Shri Saran Raj N <[email protected]>
saranrajnk
force-pushed
the
nexus-wp-feat
branch
from
b19028b to
e1db8de
Compare
saranrajnk
commented
Jan 26, 2025
|
|
||
| // osClient needs spark session to be created first to get FlintOptions initialized. | ||
| // Otherwise, we will have connection exception from EMR-S to OS. | ||
| val osClient = new OSClient(FlintSparkConf().flintOptions()) | ||
|
|
||
| // QueryResultWriter depends on sessionManager to fetch the sessionContext | ||
| val sessionManager = instantiateSessionManager(sparkSession, Some(resultIndex)) |
There was a problem hiding this comment.
Since JobOperator needs to support interactive queries, QueryResultWriter will be used. QueryResultWriterImpl, which handles the writing of query results, depends on sessionManager.
That's why sessionManager is being introduced here to satisfy this dependency (for interactive queries)
Signed-off-by: Shri Saran Raj N <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces support for FlintJob to handle all types of queries — interactive, streaming, and batch — with all data sources in warmpool mode. Additionally, FlintJob will also support non-warmpool mode for streaming and batch queries, configurable via a Spark configuration setting.
FlintJob invokes
Warmpool.scala, which in turn calls the client to continuously fetch queries for execution. The client sets various Spark configurations, such as the datasource, resultIndex, and other parameters. It also controls when to terminate the loop and stop the job. When a valid query is received, the JobOperator flow is triggered to execute the query and write the results accordingly.Changes:
getNextStatement()in a loop.JobOperatorto write the query results either to QueryResultWriter or an OpenSearch Index, depending on the job type.JobOperator.Related Issues
Check List
--signoffBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.