fix converting date in yyyyMM and yyyy formats to Unix timestamp report Invalid value for DayOfMonth (valid values 1–28/31): 0. #57234

KeeProMise · 2025-10-22T09:02:47Z

What problem does this PR solve?

Issue Number: #57233

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

KeeProMise · 2025-10-23T02:10:19Z

run buildall

doris-robot · 2025-10-23T02:47:48Z

ClickBench: Total hot run time: 29.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a301cdbd311180e9bfdf6a187b416ab9c1801355, data reload: false

query1	0.07	0.06	0.07
query2	0.11	0.06	0.06
query3	0.27	0.09	0.10
query4	1.61	0.12	0.12
query5	0.29	0.28	0.29
query6	1.19	0.67	0.69
query7	0.04	0.03	0.03
query8	0.07	0.05	0.05
query9	0.67	0.56	0.58
query10	0.64	0.64	0.62
query11	0.19	0.14	0.13
query12	0.18	0.14	0.14
query13	0.64	0.62	0.61
query14	1.02	1.03	1.08
query15	0.92	0.88	0.89
query16	0.42	0.42	0.43
query17	1.14	1.19	1.17
query18	0.24	0.22	0.22
query19	1.98	1.94	1.89
query20	0.02	0.02	0.01
query21	15.36	0.25	0.14
query22	4.89	0.08	0.06
query23	15.65	0.30	0.12
query24	2.71	0.72	0.70
query25	0.09	0.08	0.07
query26	0.18	0.15	0.15
query27	0.08	0.06	0.06
query28	5.38	1.19	0.97
query29	12.59	4.56	3.70
query30	0.30	0.17	0.13
query31	2.83	0.65	0.40
query32	3.25	0.57	0.49
query33	3.11	3.07	3.22
query34	15.87	5.22	4.55
query35	4.63	4.62	4.64
query36	0.69	0.53	0.52
query37	0.11	0.08	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.04
query40	0.18	0.15	0.16
query41	0.10	0.04	0.04
query42	0.04	0.04	0.04
query43	0.06	0.04	0.04
Total cold run time: 99.92 s
Total hot run time: 29.22 s

hello-stephen · 2025-10-23T03:27:02Z

FE UT Coverage Report

Increment line coverage 40.00% (4/10) 🎉
Increment coverage report
Complete coverage report

zclllyybb · 2025-10-23T07:34:16Z

fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateUtils.java

+        if (accessor.isSupported(ChronoField.DAY_OF_MONTH)) {
+            return accessor.get(ChronoField.DAY_OF_MONTH);
+        } else if (accessor.isSupported(ChronoField.DAY_OF_YEAR)) {
+            return accessor.get(ChronoField.DAY_OF_YEAR);


day of year is 1..366. not suitable here.

zclllyybb

could you attach an error backstrace here? By this issue I'm not very sure whether this fix is proper

zclllyybb · 2025-10-23T07:39:06Z

fe/fe-core/src/main/java/org/apache/doris/nereids/util/DateUtils.java

@@ -370,8 +370,8 @@ public static LocalDateTime getTime(DateTimeFormatter formatter, String value) {
        TemporalAccessor accessor = formatter.parse(value);
        return LocalDateTime.of(
                getOrDefault(accessor, ChronoField.YEAR),
-                getOrDefault(accessor, ChronoField.MONTH_OF_YEAR),
-                getOrDefault(accessor, ChronoField.DAY_OF_MONTH),
+                getMonthOrDefault(accessor),


By changing this, you actually changed the function str_to_date's behaviour of FE. you should add some regression cases to make sure the behaviour is same of BE and FE. could use qt_sql and testFoldConst in reg.

KeeProMise · 2025-10-23T12:27:55Z

run buildall

doris-robot · 2025-10-23T13:18:56Z

ClickBench: Total hot run time: 28.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aeb3b847d181d71020b008f9159e4e7ef4206f4c, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.08	0.09
query4	1.62	0.12	0.12
query5	0.28	0.25	0.25
query6	1.19	0.67	0.65
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.62	0.54	0.53
query10	0.59	0.58	0.58
query11	0.17	0.12	0.14
query12	0.16	0.12	0.12
query13	0.62	0.60	0.61
query14	1.02	1.03	1.02
query15	0.86	0.85	0.87
query16	0.39	0.42	0.40
query17	1.01	1.03	1.03
query18	0.22	0.20	0.20
query19	1.91	1.82	1.85
query20	0.02	0.02	0.01
query21	15.48	0.18	0.13
query22	5.15	0.07	0.06
query23	15.66	0.26	0.11
query24	3.61	0.53	0.49
query25	0.08	0.07	0.06
query26	0.14	0.13	0.14
query27	0.07	0.06	0.06
query28	4.10	1.14	0.98
query29	12.55	4.02	3.33
query30	0.28	0.17	0.11
query31	2.83	0.59	0.38
query32	3.25	0.56	0.46
query33	3.03	3.15	3.13
query34	15.84	5.19	4.59
query35	4.59	4.56	4.60
query36	0.66	0.50	0.50
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.04
query40	0.17	0.15	0.15
query41	0.08	0.04	0.03
query42	0.05	0.03	0.04
query43	0.04	0.04	0.03
Total cold run time: 99.03 s
Total hot run time: 28.02 s

hello-stephen · 2025-10-23T15:35:11Z

FE Regression Coverage Report

Increment line coverage 80.00% (8/10) 🎉
Increment coverage report
Complete coverage report

KeeProMise · 2025-10-24T02:53:37Z

run buildall

doris-robot · 2025-10-24T03:36:59Z

TPC-DS: Total hot run time: 190769 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 020acc33f7cbe7b89c12bbe5eff81d59bef5d4f7, data reload: false

query1	1068	445	397	397
query2	6581	1736	1709	1709
query3	6774	230	230	230
query4	26232	23493	23682	23493
query5	4943	682	501	501
query6	367	240	230	230
query7	4675	503	304	304
query8	331	279	267	267
query9	8764	2587	2576	2576
query10	538	376	313	313
query11	15318	15092	15604	15092
query12	203	140	134	134
query13	1739	582	449	449
query14	11803	9581	9500	9500
query15	302	210	182	182
query16	8580	709	588	588
query17	2341	805	683	683
query18	2156	440	346	346
query19	301	218	187	187
query20	142	131	127	127
query21	245	130	128	128
query22	4521	4470	4411	4411
query23	35113	33966	33854	33854
query24	8304	2434	2477	2434
query25	600	534	481	481
query26	1222	277	165	165
query27	2689	505	348	348
query28	4335	2183	2180	2180
query29	806	651	520	520
query30	296	214	199	199
query31	930	877	734	734
query32	77	70	70	70
query33	587	378	332	332
query34	803	865	522	522
query35	788	823	764	764
query36	948	986	909	909
query37	126	121	82	82
query38	3570	3601	3511	3511
query39	1467	1419	1443	1419
query40	233	128	120	120
query41	64	57	58	57
query42	119	111	106	106
query43	482	505	473	473
query44	1216	746	726	726
query45	182	185	175	175
query46	889	1002	635	635
query47	1799	1795	1732	1732
query48	392	417	326	326
query49	777	516	416	416
query50	675	694	405	405
query51	3935	3917	3838	3838
query52	108	114	108	108
query53	233	266	199	199
query54	607	603	540	540
query55	103	89	87	87
query56	306	336	309	309
query57	1193	1177	1105	1105
query58	288	284	284	284
query59	2561	2679	2531	2531
query60	379	350	328	328
query61	169	165	156	156
query62	806	742	686	686
query63	244	199	195	195
query64	4475	1221	868	868
query65	4066	3956	3937	3937
query66	1082	458	419	419
query67	15157	14932	14948	14932
query68	9678	880	607	607
query69	483	328	323	323
query70	1315	1305	1220	1220
query71	501	344	315	315
query72	6029	4866	4864	4864
query73	734	565	359	359
query74	9130	8987	8934	8934
query75	4400	3375	2887	2887
query76	3585	1158	728	728
query77	956	402	316	316
query78	9591	9740	8931	8931
query79	1997	822	596	596
query80	639	642	503	503
query81	507	264	229	229
query82	447	179	138	138
query83	295	279	263	263
query84	302	115	93	93
query85	905	499	449	449
query86	333	318	310	310
query87	3737	3765	3662	3662
query88	3459	2275	2219	2219
query89	396	330	304	304
query90	1986	236	215	215
query91	162	170	137	137
query92	88	73	73	73
query93	1210	982	649	649
query94	705	436	362	362
query95	413	326	320	320
query96	505	581	292	292
query97	2942	2963	2880	2880
query98	239	218	217	217
query99	1471	1423	1310	1310
Total cold run time: 282265 ms
Total hot run time: 190769 ms

doris-robot · 2025-10-24T03:42:06Z

ClickBench: Total hot run time: 27.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 020acc33f7cbe7b89c12bbe5eff81d59bef5d4f7, data reload: false

query1	0.05	0.05	0.05
query2	0.11	0.06	0.06
query3	0.26	0.08	0.08
query4	1.61	0.12	0.11
query5	0.28	0.26	0.24
query6	1.16	0.65	0.64
query7	0.03	0.03	0.02
query8	0.06	0.04	0.04
query9	0.64	0.54	0.52
query10	0.58	0.59	0.58
query11	0.18	0.11	0.12
query12	0.16	0.12	0.13
query13	0.63	0.61	0.61
query14	1.01	1.01	1.04
query15	0.85	0.83	0.85
query16	0.39	0.41	0.40
query17	1.04	1.04	1.01
query18	0.21	0.20	0.20
query19	1.95	1.83	1.86
query20	0.02	0.01	0.01
query21	15.44	0.21	0.13
query22	4.97	0.08	0.05
query23	15.64	0.26	0.11
query24	2.28	0.56	0.50
query25	0.08	0.07	0.08
query26	0.14	0.14	0.14
query27	0.07	0.06	0.05
query28	4.18	1.15	0.93
query29	12.59	3.98	3.27
query30	0.30	0.14	0.12
query31	2.82	0.59	0.38
query32	3.24	0.56	0.48
query33	3.03	3.05	3.07
query34	15.77	5.19	4.56
query35	4.59	4.57	4.62
query36	0.70	0.51	0.50
query37	0.10	0.06	0.06
query38	0.06	0.04	0.05
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.6 s
Total hot run time: 27.74 s

hello-stephen · 2025-10-24T04:01:23Z

FE UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-10-24T05:47:51Z

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

…amp.

KeeProMise · 2025-10-24T08:49:09Z

run buildall

doris-robot · 2025-10-24T09:37:28Z

ClickBench: Total hot run time: 28.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77516d4bde3f6b260e6751271fc729e36b93b1ce, data reload: false

query1	0.07	0.06	0.05
query2	0.10	0.05	0.06
query3	0.26	0.09	0.08
query4	1.62	0.12	0.13
query5	0.28	0.27	0.25
query6	1.19	0.67	0.68
query7	0.04	0.02	0.03
query8	0.06	0.05	0.05
query9	0.63	0.56	0.54
query10	0.59	0.60	0.60
query11	0.18	0.12	0.12
query12	0.16	0.14	0.13
query13	0.62	0.65	0.61
query14	1.03	1.02	1.03
query15	0.88	0.85	0.88
query16	0.41	0.44	0.40
query17	1.05	1.05	1.06
query18	0.24	0.21	0.21
query19	2.03	1.89	1.84
query20	0.02	0.02	0.01
query21	15.40	0.21	0.14
query22	5.02	0.07	0.06
query23	15.64	0.27	0.12
query24	2.89	1.44	0.47
query25	0.10	0.07	0.06
query26	0.18	0.15	0.14
query27	0.08	0.07	0.06
query28	4.78	1.19	0.95
query29	12.61	4.13	3.41
query30	0.29	0.15	0.11
query31	2.83	0.60	0.39
query32	3.25	0.57	0.49
query33	3.14	3.10	3.16
query34	15.88	5.18	4.59
query35	4.64	4.62	4.58
query36	0.71	0.54	0.50
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.15	0.14
query41	0.08	0.04	0.04
query42	0.04	0.04	0.03
query43	0.05	0.05	0.03
Total cold run time: 99.46 s
Total hot run time: 28.21 s

hello-stephen · 2025-10-24T11:49:40Z

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

KeeProMise · 2025-10-24T11:59:59Z

Hi @zclllyybb @yiguolei If you have time, could you please take a look at this PR? The failing tests now pass on my local machine.

zclllyybb · 2025-10-25T19:06:06Z

Hi @zclllyybb @yiguolei If you have time, could you please take a look at this PR? The failing tests now pass on my local machine.

Hi @KeeProMise , thanks for your contribution! In this PR you actually modified the logic of function str_to_date, right?
so,

could you add a stacktrace or something to explain how the functions' behaviour of the two weird format input affects the mv? @zddr @seawinde do you have more information?
select STR_TO_DATE('202501', '%Y%m'); is also not supported by MySQL. so I'm not sure if this fix is proper. same to the point 1, I want firstly figure out if there's some way more directly to fix your problem.
if we decide to accept this patch, you should add some regression testcases using testFoldConst to make sure the behaviour has no diff between BE and FE.

KeeProMise · 2025-10-27T12:39:57Z

@zclllyybb stacktrace

Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Invalid value for DayOfMonth (valid values 1 - 28/31): 0
    ... 18 more
Caused by: java.time.DateTimeException: Invalid value for DayOfMonth (valid values 1 - 28/31): 0
    at java.time.temporal.ValueRange.checkValidValue(ValueRange.java:319) ~[?:?]
    at java.time.temporal.ChronoField.checkValidValue(ChronoField.java:718) ~[?:?]
    at java.time.LocalDate.of(LocalDate.java:272) ~[?:?]
    at java.time.LocalDateTime.of(LocalDateTime.java:363) ~[?:?]
    at org.apache.doris.nereids.util.DateUtils.getTime(DateUtils.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.expressions.functions.executable.DateTimeExtractAndTransform.strToDate(DateTimeExtractAndTransform.java:695) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.mtmv.MTMVUtil.getExprTimeSec(MTMVUtil.java:129) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.catalog.ListPartitionItem.isGreaterThanSpecifiedTime(ListPartitionItem.java:130) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.mtmv.MTMVRelatedPartitionDescSyncLimitGenerator.apply(MTMVRelatedPartitionDescSyncLimitGenerator.java:62) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.mtmv.MTMVPartitionUtil.generateRelatedPartitionDescs(MTMVPartitionUtil.java:184) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.mtmv.MTMVPartitionUtil.getPartitionDescsByRelatedTable(MTMVPartitionUtil.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.plans.commands.info.CreateMTMVInfo.generatePartitionDesc(CreateMTMVInfo.java:333) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.plans.commands.info.CreateMTMVInfo.analyzeQuery(CreateMTMVInfo.java:273) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.plans.commands.info.CreateMTMVInfo.analyze(CreateMTMVInfo.java:163) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.nereids.trees.plans.commands.CreateMTMVCommand.run(CreateMTMVCommand.java:51) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:753) ~[doris-fe.jar:1.2-SNAPSHOT]
    ... 17 more

yiguolei added the dev/4.0.x label Oct 23, 2025

zclllyybb self-assigned this Oct 23, 2025

zclllyybb reviewed Oct 23, 2025

View reviewed changes

zclllyybb requested changes Oct 23, 2025

View reviewed changes

zclllyybb requested a review from Jibing-Li October 23, 2025 07:40

KeeProMise added 5 commits October 24, 2025 15:45

fix Support converting date in yyyyMM and yyyy formats to Unix timest…

f619e81

…amp.

fix checkstyle

ee11b2d

fix ut

f3eb161

fix ut

0229d6f

add regression-test

77516d4

KeeProMise force-pushed the 57233 branch from 020acc3 to 77516d4 Compare October 24, 2025 08:48

zclllyybb requested a review from morrySnow October 27, 2025 06:12

Uh oh!

fix converting date in yyyyMM and yyyy formats to Unix timestamp report Invalid value for DayOfMonth (valid values 1–28/31): 0. #57234

Are you sure you want to change the base?

fix converting date in yyyyMM and yyyy formats to Unix timestamp report Invalid value for DayOfMonth (valid values 1–28/31): 0. #57234

Conversation

KeeProMise commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

KeeProMise commented Oct 23, 2025

Uh oh!

doris-robot commented Oct 23, 2025

Uh oh!

hello-stephen commented Oct 23, 2025

FE UT Coverage Report

Uh oh!

zclllyybb Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

KeeProMise Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

zclllyybb left a comment

Choose a reason for hiding this comment

Uh oh!

zclllyybb Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

KeeProMise Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

KeeProMise commented Oct 23, 2025

Uh oh!

doris-robot commented Oct 23, 2025

Uh oh!

hello-stephen commented Oct 23, 2025

FE Regression Coverage Report

Uh oh!

KeeProMise commented Oct 24, 2025

Uh oh!

doris-robot commented Oct 24, 2025

Uh oh!

doris-robot commented Oct 24, 2025

Uh oh!

hello-stephen commented Oct 24, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 24, 2025

FE Regression Coverage Report

Uh oh!

KeeProMise commented Oct 24, 2025

Uh oh!

doris-robot commented Oct 24, 2025

Uh oh!

hello-stephen commented Oct 24, 2025

FE Regression Coverage Report

Uh oh!

KeeProMise commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zclllyybb commented Oct 25, 2025

Uh oh!

KeeProMise commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

KeeProMise commented Oct 22, 2025 •

edited

Loading

KeeProMise commented Oct 24, 2025 •

edited

Loading

KeeProMise commented Oct 27, 2025 •

edited

Loading