[FLINK-35856] [table] Upgrade Calcite version to 1.37.0 #26895

michalstutzmann · 2025-08-12T11:47:54Z

What is the purpose of the change

Upgrades Calcite to version 1.37.0.

This PR covers only Calcite upgrade without the implementation of lambda expressions in SQL (CALCITE-3679) on the Flink side.

Brief change log

Upgrade of Calcite to version 1.37
All necessary code adjustments.

Verifying this change

Existing tests.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes - CALCITE-6138 - TIME WITH TIME ZONE/TIMESTAMP WITH TIME ZONE)
If yes, how is the feature documented? (not documented)

flinkbot · 2025-08-12T11:53:12Z

CI report:

64ead2b Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

snuyanzin

Thank you for the hard work @michalstutzmann
the PR already looks good
I left some comments

snuyanzin · 2025-08-12T22:56:23Z

.../scala/org/apache/flink/table/planner/plan/rules/logical/subquery/SubQuerySemiJoinTest.scala

-
-    val sqlQuery2 = "SELECT MAX(a) FROM x GROUP BY 1 HAVING EXISTS (SELECT 1 FROM y WHERE d < b)"
-    util.verifyRelPlanNotExpected(sqlQuery2, "joinType=[semi]")
+    util.verifyRelPlanNotExpected(sqlQuery, "joinType=[semi]")


what is the reason for such change in tests

snuyanzin · 2025-08-12T22:56:50Z

flink-table/pom.xml

-		<calcite.version>1.36.0</calcite.version>
-		<!-- Calcite 1.36.0 depends on 3.1.9,
+		<calcite.version>1.37.0</calcite.version>
+		<!-- Calcite 1.37.0 depends on 3.1.9,


this comment is no longer true
Calcite 1.37 depends on janino 3.1.10
https://central.sonatype.com/artifact/org.apache.calcite/calcite-core/1.37.0/dependencies

not sure, may be the version bump could be done separately if it doesn't block

Have corrected the comment. 1.37 depends already on 3.1.11 which is causing the tests to fail but with 3.1.10 it seems to work. I'll have another look. If bumping to 3.1.11 is a lot of work (lot of changes) I agree it's probably better to do it separately.

Update: trying out building with 3.1.11.

Janino 3.1.11 has changed the way the compiler factory is loaded - has switched to Java's ServiceLoader. The e2e tests were failing due class loading issues - the META-INF/services from Janino's jar could not be found by ServiceLoader. Adding the entry to table planner's jar resolves the issue. This is the simplest solution I've found so far. Have upgraded Janino to the latest patch version 3.1.12.

snuyanzin · 2025-08-12T22:58:58Z

...ble-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/ValuesTest.xml

+Calc(select=[1 AS a, b, c])
+- Values(tuples=[[{ 1, 2, 3 }]])


I wonder why do we start having extra calc here and for other tests?

It's caused by CALCITE-6044. We could revert to the 1.36 behaviour here.

snuyanzin · 2025-08-12T23:02:49Z

flink-table/flink-sql-parser/src/main/codegen/templates/Parser.jj

+    <TIME> { s = span(); } <WITH> ( <LOCAL> { local = true; } )? <TIME> <ZONE> p = SimpleStringLiteral() {
+        return SqlLiteral.createUnknown("TIME WITH " + (local ? "LOCAL " : "") + "TIME ZONE", p, s.end(this));
+    }
+|
+    LOOKAHEAD(2)
+    <TIMESTAMP> { s = span(); } <WITH> ( <LOCAL> { local = true; } )? <TIME> <ZONE> p = SimpleStringLiteral() {
+        return SqlLiteral.createUnknown("TIMESTAMP WITH " + (local ? "LOCAL " : "") + "TIME ZONE", p, s.end(this));


is it now also supported for Flink?

snuyanzin · 2025-08-12T23:03:30Z

flink-table/flink-sql-parser/src/main/codegen/templates/Parser.jj

+/**
+ * Parses a lambda expression.
+ */
+SqlNode LambdaExpression() :


can we have a dummy test showing that it fails somewhere on validation rather than in runtime?

davidradl · 2025-08-13T12:55:05Z

@snuyanzin @michalstutzmann I see changes in the tests around the SQL and am worried whether this could mean there is a backwards compatibility issue in some way. For example would restoring savepoints across the Calcite version change, work? Is there a test for this? I assume a more efficient optimization at a higher Calcite level does not cause a compatibility issue - though the explain output may change. It would be great if you could reassure me these worries are unfounded.

davidradl · 2025-08-15T13:15:01Z

@snuyanzin @michalstutzmann I see changes in the tests around the SQL and am worried whether this could mean there is a backwards compatibility issue in some way. For example would restoring savepoints across the Calcite version change, work? Is there a test for this? I assume a more efficient optimization at a higher Calcite level does not cause a compatibility issue - though the explain output may change. It would be great if you could reassure me these worries are unfounded.

@michalstutzmann I see you have thumbs up this question I am not sure what you mean by this. I see the docs say In other words: Savepoints are only supported if both the query and the Flink version remain constant. I assume this statement also effects Calcite version changes. Do you think this change will mean Savepoints are not supported across this change?

michalstutzmann · 2025-08-15T13:31:44Z

@snuyanzin @michalstutzmann I see changes in the tests around the SQL and am worried whether this could mean there is a backwards compatibility issue in some way. For example would restoring savepoints across the Calcite version change, work? Is there a test for this? I assume a more efficient optimization at a higher Calcite level does not cause a compatibility issue - though the explain output may change. It would be great if you could reassure me these worries are unfounded.

@michalstutzmann I see you have thumbs up this question I am not sure what you mean by this. I see the docs say In other words: Savepoints are only supported if both the query and the Flink version remain constant. I assume this statement also effects Calcite version changes. Do you think this change will mean Savepoints are not supported across this change?

@davidradl I will check all those things soon and come back to you. Currently on the Janino upgrade and failing tests there (see earlier comment).

davidradl · 2025-08-15T15:16:08Z

@davidradl I will check all those things soon and come back to you. Currently on the Janino upgrade and failing tests there (see earlier comment).
@michalstutzmann thank you so much :-)

snuyanzin · 2025-08-15T21:10:04Z

I see changes in the tests around the SQL and am worried whether this could mean there is a backwards compatibility issue in some way. For example would restoring savepoints across the Calcite version change, work? Is there a test for this? I assume a more efficient optimization at a higher Calcite level does not cause a compatibility issue - though the explain output may change. It would be great if you could reassure me these worries are unfounded.

if a change (not only Calcite upgrade) leads to change (not only description change) in CompiledPlan store/restore tests then it is most probably a breaking change. If a change leads to change in optimized plan tests then it is at least questionable... Of course our tests do not cover all the cases and could miss something
also more details here https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489

Upgrade Calcite version to 1.37.0

57c2fec

Michal Stutzmann added 3 commits August 12, 2025 16:21

Format

ff045c1

Format

a4c4fb7

Update versions for licence checks

a443034

snuyanzin reviewed Aug 12, 2025

View reviewed changes

Michal Stutzmann added 5 commits August 14, 2025 06:35

Correct comment about Janino version

e25fe85

Try Janino 3.1.11

3c90fe0

Add Janino to NOTICE file for licence checks

0cd5508

Fix Janino licence checks

5ce11ca

Try Janino 3.1.12

02bc92f

Add Janino's compiler factory service to META-INF/services

64ead2b

		Calc(select=[1 AS a, b, c])
		+- Values(tuples=[[{ 1, 2, 3 }]])

[FLINK-35856] [table] Upgrade Calcite version to 1.37.0 #26895

Are you sure you want to change the base?

[FLINK-35856] [table] Upgrade Calcite version to 1.37.0 #26895

Uh oh!

Conversation

michalstutzmann commented Aug 12, 2025

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

snuyanzin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidradl commented Aug 13, 2025

Uh oh!

davidradl commented Aug 15, 2025

Uh oh!

michalstutzmann commented Aug 15, 2025

Uh oh!

davidradl commented Aug 15, 2025

Uh oh!

snuyanzin commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

flinkbot commented Aug 12, 2025 •

edited

Loading

snuyanzin commented Aug 15, 2025 •

edited

Loading