Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](metrics) Unify metrics of thread pool #43144

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Nov 2, 2024

What problem does this PR solve?

Add metrics for all thread pool, more specifically, for all ThreadPool objects.
All thread pool will have following metrics:

  1. thread_pool_active_threads
  2. thread_pool_queue_size
  3. thread_pool_max_queue_size
  4. thread_pool_max_threads
  5. task_execution_time_ns_avg_in_last_1000_times
  6. task_wait_worker_ns_avg_in_last_1000_times

A new class IntervalHistogramStat is created for interval histogram calculation.

Metrics is updated by hook method when they are needed by prometheus.

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.
  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/exec/scan/scanner_scheduler.cpp Outdated Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25967)
Line Coverage: 28.99% (81627/281564)
Region Coverage: 28.25% (42144/149158)
Branch Coverage: 24.83% (21381/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b36415796af4881788f26f82f81791a16e8e4608_b36415796af4881788f26f82f81791a16e8e4608/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as draft November 4, 2024 00:59
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9820/25967)
Line Coverage: 28.99% (81616/281564)
Region Coverage: 28.25% (42141/149158)
Branch Coverage: 24.83% (21379/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b36415796af4881788f26f82f81791a16e8e4608_b36415796af4881788f26f82f81791a16e8e4608/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (9821/25970)
Line Coverage: 28.97% (81596/281672)
Region Coverage: 28.24% (42131/149203)
Branch Coverage: 24.82% (21381/86140)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7eb8642f9a587895a5c757f61d356507495daed1_7eb8642f9a587895a5c757f61d356507495daed1/report/index.html

@zhiqiang-hhhh
Copy link
Contributor Author

image

Fail of test has nothing to do with this pr.

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review November 4, 2024 12:51
wangbo
wangbo previously approved these changes Nov 6, 2024
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/util/interval_histogram.h Show resolved Hide resolved
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh changed the title [opt](metrics) More metrics for scanner [feat](metrics) Unify metrics of thread pool Dec 17, 2024
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh zhiqiang-hhhh marked this pull request as ready for review December 19, 2024 12:29
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

1 similar comment
@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33416 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bfdd0d7ae28005907d17e584a2e1b9e17eceb47, data reload: false

------ Round 1 ----------------------------------
q1	18789	6206	6049	6049
q2	2047	306	165	165
q3	11310	1233	748	748
q4	10197	862	433	433
q5	7523	2220	1982	1982
q6	203	180	151	151
q7	898	753	604	604
q8	9324	1374	1217	1217
q9	5411	4995	4958	4958
q10	6762	2334	1846	1846
q11	486	289	271	271
q12	356	354	224	224
q13	17983	3797	3122	3122
q14	245	246	230	230
q15	552	510	490	490
q16	627	636	619	619
q17	584	865	337	337
q18	9256	6949	6812	6812
q19	4680	987	570	570
q20	295	311	181	181
q21	2797	2351	2088	2088
q22	374	344	319	319
Total cold run time: 110699 ms
Total hot run time: 33416 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6390	6189	6142	6142
q2	235	325	233	233
q3	2302	2652	2340	2340
q4	1450	1823	1382	1382
q5	4462	4999	4921	4921
q6	186	184	143	143
q7	2107	1938	1839	1839
q8	2584	2760	2680	2680
q9	7327	7324	7182	7182
q10	2994	3372	2826	2826
q11	585	524	495	495
q12	684	759	603	603
q13	3366	3774	3163	3163
q14	297	296	297	296
q15	553	519	516	516
q16	641	699	654	654
q17	1204	1736	1241	1241
q18	7657	7474	7499	7474
q19	827	981	1144	981
q20	2061	2070	1921	1921
q21	5641	5255	5067	5067
q22	616	638	588	588
Total cold run time: 54169 ms
Total hot run time: 52687 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196400 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bfdd0d7ae28005907d17e584a2e1b9e17eceb47, data reload: false

query1	1303	962	926	926
query2	6409	2329	2400	2329
query3	10986	4604	4779	4604
query4	33597	23809	23531	23531
query5	3696	597	476	476
query6	273	200	197	197
query7	3980	481	310	310
query8	295	237	227	227
query9	9659	2695	2698	2695
query10	474	300	241	241
query11	17804	15141	15294	15141
query12	157	104	100	100
query13	1568	530	408	408
query14	9691	6918	7006	6918
query15	243	241	192	192
query16	7993	607	494	494
query17	1520	732	560	560
query18	2125	408	316	316
query19	216	192	169	169
query20	129	120	120	120
query21	200	127	117	117
query22	4735	4610	4495	4495
query23	34801	33611	33799	33611
query24	6488	2328	2360	2328
query25	476	492	397	397
query26	716	278	156	156
query27	2001	463	334	334
query28	5866	2497	2434	2434
query29	581	550	420	420
query30	212	184	150	150
query31	995	999	838	838
query32	67	59	57	57
query33	474	352	289	289
query34	762	864	520	520
query35	800	885	767	767
query36	981	1047	946	946
query37	122	104	77	77
query38	4284	4444	4147	4147
query39	1532	1484	1477	1477
query40	217	114	102	102
query41	49	47	44	44
query42	125	114	114	114
query43	534	550	506	506
query44	1330	819	808	808
query45	189	179	176	176
query46	885	1046	648	648
query47	2037	2030	1994	1994
query48	385	421	335	335
query49	725	485	413	413
query50	627	686	396	396
query51	7299	7078	7148	7078
query52	105	102	101	101
query53	228	249	189	189
query54	485	495	405	405
query55	86	81	78	78
query56	275	264	262	262
query57	1288	1239	1180	1180
query58	236	226	236	226
query59	3364	3450	3266	3266
query60	288	273	261	261
query61	111	109	114	109
query62	870	834	779	779
query63	223	199	204	199
query64	2783	1038	684	684
query65	3302	3249	3290	3249
query66	780	418	308	308
query67	16430	15735	15456	15456
query68	8911	799	542	542
query69	480	291	260	260
query70	1228	1139	1188	1139
query71	436	286	252	252
query72	6003	3790	3979	3790
query73	657	745	363	363
query74	9974	9347	8999	8999
query75	4642	3143	2635	2635
query76	4116	1171	772	772
query77	769	358	277	277
query78	10301	10176	9438	9438
query79	3070	887	589	589
query80	721	502	431	431
query81	481	266	226	226
query82	624	147	126	126
query83	177	168	146	146
query84	254	82	74	74
query85	780	392	298	298
query86	349	316	301	301
query87	4572	4448	4496	4448
query88	4612	2196	2197	2196
query89	407	325	294	294
query90	1819	185	183	183
query91	137	132	107	107
query92	61	54	50	50
query93	1541	861	524	524
query94	653	388	287	287
query95	329	265	242	242
query96	487	597	278	278
query97	2754	2793	2620	2620
query98	225	196	197	196
query99	1676	1576	1437	1437
Total cold run time: 296713 ms
Total hot run time: 196400 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bfdd0d7ae28005907d17e584a2e1b9e17eceb47, data reload: false

query1	0.03	0.05	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.63	0.11	0.11
query5	0.39	0.43	0.41
query6	1.15	0.66	0.65
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.51
query10	0.55	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.83	2.86	2.80
query15	0.89	0.82	0.83
query16	0.38	0.39	0.39
query17	1.09	1.06	1.02
query18	0.23	0.20	0.19
query19	1.90	1.75	1.95
query20	0.01	0.01	0.01
query21	15.36	0.90	0.59
query22	0.76	0.72	0.62
query23	15.40	1.45	0.61
query24	3.34	0.60	1.16
query25	0.19	0.17	0.13
query26	0.31	0.14	0.13
query27	0.06	0.05	0.04
query28	13.56	1.54	1.04
query29	12.56	3.99	3.28
query30	0.25	0.10	0.06
query31	2.83	0.59	0.37
query32	3.24	0.53	0.46
query33	3.04	3.14	3.15
query34	16.85	5.10	4.50
query35	4.48	4.45	4.45
query36	0.63	0.48	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.25 s
Total hot run time: 30.99 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32530 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec0702ca7597325d50865d4b3ee252a050efe668, data reload: false

------ Round 1 ----------------------------------
q1	17590	6147	6035	6035
q2	2046	300	166	166
q3	10422	1255	697	697
q4	10203	857	435	435
q5	7525	2160	1947	1947
q6	205	181	149	149
q7	896	737	626	626
q8	9225	1357	1189	1189
q9	5184	4913	4985	4913
q10	6761	2343	1864	1864
q11	469	283	264	264
q12	351	363	220	220
q13	17761	3521	2932	2932
q14	228	243	216	216
q15	554	509	505	505
q16	633	613	579	579
q17	562	842	325	325
q18	7173	6443	6440	6440
q19	1490	965	554	554
q20	315	319	187	187
q21	2852	2176	1974	1974
q22	359	328	313	313
Total cold run time: 102804 ms
Total hot run time: 32530 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6202	6167	6193	6167
q2	231	336	236	236
q3	2237	2612	2347	2347
q4	1378	1788	1366	1366
q5	4323	4729	4716	4716
q6	191	172	141	141
q7	2132	1941	1838	1838
q8	2583	2797	2692	2692
q9	7299	7286	7246	7246
q10	3051	3305	2806	2806
q11	581	507	500	500
q12	667	737	576	576
q13	3347	3747	3083	3083
q14	286	284	278	278
q15	577	527	500	500
q16	644	686	622	622
q17	1181	1713	1264	1264
q18	7948	7463	7234	7234
q19	836	891	1121	891
q20	1945	1970	1833	1833
q21	5606	5156	4787	4787
q22	599	668	578	578
Total cold run time: 53844 ms
Total hot run time: 51701 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190511 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec0702ca7597325d50865d4b3ee252a050efe668, data reload: false

query1	974	388	364	364
query2	6535	2443	2419	2419
query3	6712	212	212	212
query4	34652	23660	23573	23573
query5	4428	633	457	457
query6	292	203	186	186
query7	4626	511	310	310
query8	316	259	243	243
query9	9694	2744	2727	2727
query10	483	326	264	264
query11	18846	15469	15490	15469
query12	166	102	101	101
query13	1654	538	435	435
query14	11897	7390	8002	7390
query15	234	198	196	196
query16	8052	589	432	432
query17	1556	756	558	558
query18	2092	398	294	294
query19	233	176	148	148
query20	126	114	110	110
query21	209	123	103	103
query22	4342	4502	4302	4302
query23	35851	33348	33281	33281
query24	6311	2340	2330	2330
query25	437	437	392	392
query26	1098	256	157	157
query27	2058	468	343	343
query28	5293	2435	2422	2422
query29	549	506	425	425
query30	229	189	155	155
query31	1005	908	839	839
query32	82	63	58	58
query33	491	361	307	307
query34	758	858	525	525
query35	819	833	765	765
query36	1031	1053	964	964
query37	129	103	78	78
query38	4418	4113	4347	4113
query39	1525	1438	1435	1435
query40	216	122	102	102
query41	47	43	44	43
query42	118	105	107	105
query43	529	535	504	504
query44	1320	816	798	798
query45	190	178	168	168
query46	881	1046	664	664
query47	1914	1896	1841	1841
query48	392	414	338	338
query49	795	477	395	395
query50	650	658	396	396
query51	7199	7326	7131	7131
query52	101	105	93	93
query53	233	263	185	185
query54	475	486	404	404
query55	78	79	79	79
query56	256	268	235	235
query57	1189	1200	1134	1134
query58	246	223	231	223
query59	3090	3264	3105	3105
query60	277	280	268	268
query61	117	111	106	106
query62	871	792	737	737
query63	236	197	200	197
query64	4636	1101	770	770
query65	3341	3253	3244	3244
query66	1081	441	336	336
query67	15880	16000	15591	15591
query68	9145	766	518	518
query69	457	294	262	262
query70	1220	1154	1145	1145
query71	427	295	261	261
query72	6160	2747	4002	2747
query73	679	796	355	355
query74	9698	9199	8936	8936
query75	4554	3055	2655	2655
query76	4282	1213	790	790
query77	936	371	278	278
query78	10281	10237	9317	9317
query79	3350	904	600	600
query80	661	530	428	428
query81	485	282	239	239
query82	650	157	129	129
query83	198	164	146	146
query84	286	95	91	91
query85	814	368	319	319
query86	355	317	280	280
query87	4597	4689	4359	4359
query88	4277	2256	2224	2224
query89	424	346	303	303
query90	1937	204	198	198
query91	131	137	145	137
query92	66	62	53	53
query93	1119	913	531	531
query94	660	389	293	293
query95	333	261	257	257
query96	491	620	280	280
query97	2730	2830	2664	2664
query98	243	206	198	198
query99	1801	1600	1444	1444
Total cold run time: 300008 ms
Total hot run time: 190511 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec0702ca7597325d50865d4b3ee252a050efe668, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.04
query3	0.23	0.08	0.07
query4	1.61	0.11	0.11
query5	0.40	0.41	0.43
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.52
query10	0.58	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.61	0.60
query14	2.81	2.86	2.72
query15	0.89	0.82	0.83
query16	0.36	0.37	0.37
query17	1.04	1.03	1.01
query18	0.24	0.21	0.21
query19	1.90	1.82	2.02
query20	0.01	0.01	0.01
query21	15.36	0.89	0.56
query22	0.75	0.75	0.63
query23	15.37	1.46	0.52
query24	3.02	1.52	1.46
query25	0.14	0.24	0.14
query26	0.19	0.14	0.14
query27	0.04	0.05	0.05
query28	14.17	1.53	1.05
query29	12.62	4.01	3.39
query30	0.24	0.09	0.07
query31	2.81	0.60	0.39
query32	3.24	0.54	0.46
query33	3.13	3.13	3.09
query34	16.81	5.10	4.47
query35	4.54	4.53	4.52
query36	0.64	0.50	0.51
query37	0.10	0.07	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.45 s
Total hot run time: 31.93 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32784 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b278524cd9b7b8e76964a363f16859aa096fe38, data reload: false

------ Round 1 ----------------------------------
q1	17599	6233	6061	6061
q2	2052	308	173	173
q3	10411	1261	722	722
q4	10216	870	440	440
q5	7526	2166	1985	1985
q6	203	179	149	149
q7	893	771	604	604
q8	9229	1341	1166	1166
q9	5255	4947	4923	4923
q10	6749	2295	1860	1860
q11	475	282	268	268
q12	351	361	227	227
q13	17754	3603	3014	3014
q14	235	236	221	221
q15	579	513	495	495
q16	652	634	600	600
q17	563	848	331	331
q18	6999	6529	6462	6462
q19	1233	982	538	538
q20	316	324	187	187
q21	2824	2220	2034	2034
q22	367	326	324	324
Total cold run time: 102481 ms
Total hot run time: 32784 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6236	6171	6187	6171
q2	239	326	234	234
q3	2267	2653	2322	2322
q4	1418	1811	1327	1327
q5	4324	4790	4616	4616
q6	182	174	138	138
q7	1965	1883	1714	1714
q8	2499	2676	2599	2599
q9	6957	6892	6882	6882
q10	2959	3238	2739	2739
q11	578	500	499	499
q12	637	702	566	566
q13	3264	3644	2942	2942
q14	285	303	258	258
q15	551	505	496	496
q16	644	686	626	626
q17	1159	1677	1210	1210
q18	7424	7159	6955	6955
q19	820	1104	1029	1029
q20	1962	1992	1867	1867
q21	5482	5140	5089	5089
q22	591	645	597	597
Total cold run time: 52443 ms
Total hot run time: 50876 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190743 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b278524cd9b7b8e76964a363f16859aa096fe38, data reload: false

query1	973	374	367	367
query2	6524	2366	2465	2366
query3	6713	212	210	210
query4	33642	23965	23647	23647
query5	4991	628	471	471
query6	271	206	181	181
query7	4622	492	299	299
query8	297	238	243	238
query9	9594	2710	2706	2706
query10	491	310	255	255
query11	18587	15314	15199	15199
query12	160	111	102	102
query13	1708	540	414	414
query14	11378	6783	6823	6783
query15	238	198	192	192
query16	8035	654	431	431
query17	1551	735	562	562
query18	2116	410	301	301
query19	219	186	158	158
query20	131	114	113	113
query21	207	126	105	105
query22	4468	4554	4309	4309
query23	34494	33262	34811	33262
query24	7070	2248	2239	2239
query25	452	437	386	386
query26	1043	251	147	147
query27	2279	454	337	337
query28	5587	2451	2385	2385
query29	527	517	409	409
query30	226	188	173	173
query31	967	892	812	812
query32	89	59	58	58
query33	501	337	292	292
query34	746	834	511	511
query35	801	807	753	753
query36	1025	1060	937	937
query37	117	99	76	76
query38	4163	4070	4299	4070
query39	1485	1463	1448	1448
query40	209	117	102	102
query41	46	46	47	46
query42	118	100	101	100
query43	536	548	487	487
query44	1262	807	797	797
query45	182	168	169	168
query46	860	1035	641	641
query47	1950	1951	1896	1896
query48	386	392	332	332
query49	770	467	398	398
query50	619	641	406	406
query51	7202	7421	7022	7022
query52	108	102	90	90
query53	216	257	184	184
query54	477	474	400	400
query55	80	77	75	75
query56	249	266	257	257
query57	1202	1199	1136	1136
query58	242	231	223	223
query59	3150	3171	3041	3041
query60	268	266	255	255
query61	116	113	111	111
query62	922	810	761	761
query63	224	186	191	186
query64	4511	996	635	635
query65	3297	3194	3230	3194
query66	854	426	317	317
query67	16006	15886	15397	15397
query68	9815	757	523	523
query69	483	290	257	257
query70	1178	1187	1131	1131
query71	453	281	253	253
query72	5838	3806	3840	3806
query73	832	751	369	369
query74	10181	9395	9022	9022
query75	4613	3143	2666	2666
query76	5669	1249	759	759
query77	1022	360	269	269
query78	10279	10205	11553	10205
query79	3691	812	588	588
query80	651	535	431	431
query81	480	269	234	234
query82	232	140	120	120
query83	197	169	148	148
query84	276	99	66	66
query85	813	365	298	298
query86	350	318	298	298
query87	4705	4757	4411	4411
query88	3447	2278	2207	2207
query89	392	321	300	300
query90	2067	188	190	188
query91	138	134	107	107
query92	72	55	55	55
query93	940	730	551	551
query94	698	400	295	295
query95	328	259	251	251
query96	488	602	276	276
query97	2903	2810	2689	2689
query98	222	195	201	195
query99	1678	1611	1414	1414
Total cold run time: 299681 ms
Total hot run time: 190743 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b278524cd9b7b8e76964a363f16859aa096fe38, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.06
query4	1.61	0.11	0.10
query5	0.42	0.40	0.40
query6	1.14	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.52	0.50
query10	0.55	0.57	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.10
query13	0.61	0.61	0.59
query14	2.70	2.73	2.73
query15	0.88	0.81	0.84
query16	0.38	0.38	0.38
query17	0.99	1.06	1.00
query18	0.23	0.19	0.21
query19	1.97	1.79	2.03
query20	0.01	0.01	0.01
query21	15.36	0.89	0.58
query22	0.76	0.84	0.77
query23	15.18	1.40	0.56
query24	2.92	1.36	1.17
query25	0.24	0.16	0.18
query26	0.23	0.14	0.13
query27	0.05	0.06	0.05
query28	14.14	1.54	1.05
query29	12.58	3.92	3.31
query30	0.26	0.08	0.06
query31	2.84	0.61	0.38
query32	3.22	0.54	0.46
query33	3.18	3.12	3.03
query34	16.50	5.20	4.53
query35	4.55	4.51	4.52
query36	0.65	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.12
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.9 s
Total hot run time: 31.6 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32577 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6061f3763916c27f0240d691ffc5a8b02210f680, data reload: false

------ Round 1 ----------------------------------
q1	17579	6390	6049	6049
q2	2045	312	176	176
q3	10468	1245	749	749
q4	10221	876	440	440
q5	7516	2195	1968	1968
q6	201	183	153	153
q7	909	758	606	606
q8	9222	1355	1136	1136
q9	5226	4957	4972	4957
q10	6760	2335	1886	1886
q11	476	279	255	255
q12	349	368	215	215
q13	17761	3620	2988	2988
q14	238	239	214	214
q15	557	499	495	495
q16	636	623	580	580
q17	578	861	331	331
q18	7099	6362	6364	6362
q19	1473	977	560	560
q20	313	316	186	186
q21	2882	2165	1965	1965
q22	357	338	306	306
Total cold run time: 102866 ms
Total hot run time: 32577 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6317	6250	6244	6244
q2	230	328	235	235
q3	2280	2671	2331	2331
q4	1497	1876	1415	1415
q5	4365	4786	4810	4786
q6	191	181	140	140
q7	2114	2009	1838	1838
q8	2702	2820	2670	2670
q9	7322	7289	7335	7289
q10	3094	3406	2832	2832
q11	583	520	484	484
q12	646	766	629	629
q13	3434	3802	3172	3172
q14	287	312	290	290
q15	574	529	510	510
q16	643	691	644	644
q17	1238	1744	1268	1268
q18	7638	7561	7475	7475
q19	844	1156	1108	1108
q20	1969	2038	1902	1902
q21	5725	5212	5090	5090
q22	623	625	589	589
Total cold run time: 54316 ms
Total hot run time: 52941 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.90% (10122/26023)
Line Coverage: 29.89% (85542/286149)
Region Coverage: 29.01% (43698/150633)
Branch Coverage: 25.54% (22296/87308)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6061f3763916c27f0240d691ffc5a8b02210f680_6061f3763916c27f0240d691ffc5a8b02210f680/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196422 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6061f3763916c27f0240d691ffc5a8b02210f680, data reload: false

query1	1303	962	928	928
query2	6511	2404	2184	2184
query3	11119	4722	4764	4722
query4	33108	23672	23406	23406
query5	4241	615	448	448
query6	288	197	187	187
query7	3991	485	303	303
query8	307	255	238	238
query9	9469	2712	2727	2712
query10	453	310	248	248
query11	17957	15291	15181	15181
query12	153	109	104	104
query13	1598	535	419	419
query14	9786	6704	6854	6704
query15	273	216	204	204
query16	7460	660	450	450
query17	1570	774	603	603
query18	1993	384	322	322
query19	198	186	160	160
query20	122	136	114	114
query21	197	136	113	113
query22	4826	4746	4483	4483
query23	35429	34352	33845	33845
query24	7614	2247	2280	2247
query25	510	457	406	406
query26	779	273	153	153
query27	2746	483	327	327
query28	6224	2502	2470	2470
query29	642	560	432	432
query30	211	193	162	162
query31	964	988	864	864
query32	81	60	56	56
query33	515	356	291	291
query34	757	866	515	515
query35	823	834	752	752
query36	1032	1088	951	951
query37	118	103	74	74
query38	4367	4371	4057	4057
query39	1520	1620	1507	1507
query40	208	114	105	105
query41	44	44	42	42
query42	113	100	106	100
query43	529	527	501	501
query44	1367	827	831	827
query45	185	178	175	175
query46	894	1052	665	665
query47	2013	2019	1993	1993
query48	389	441	328	328
query49	725	484	396	396
query50	645	666	398	398
query51	7522	7229	7292	7229
query52	109	97	90	90
query53	232	251	184	184
query54	492	491	406	406
query55	83	82	83	82
query56	266	261	258	258
query57	1254	1267	1160	1160
query58	234	222	227	222
query59	3182	3200	3085	3085
query60	294	260	282	260
query61	109	111	112	111
query62	908	839	780	780
query63	232	193	187	187
query64	3707	1032	670	670
query65	3325	3233	3271	3233
query66	780	475	304	304
query67	16156	15885	15670	15670
query68	9728	763	525	525
query69	484	283	249	249
query70	1230	1136	1142	1136
query71	422	295	251	251
query72	6247	3895	3915	3895
query73	666	770	364	364
query74	10309	8949	9224	8949
query75	4425	3150	2667	2667
query76	4761	1200	772	772
query77	843	355	279	279
query78	10199	10272	9501	9501
query79	4683	893	576	576
query80	730	513	411	411
query81	488	270	224	224
query82	455	155	122	122
query83	215	157	144	144
query84	283	86	74	74
query85	808	369	312	312
query86	346	306	310	306
query87	4529	4476	4274	4274
query88	3584	2253	2241	2241
query89	444	339	290	290
query90	1979	188	186	186
query91	135	137	105	105
query92	64	54	60	54
query93	3015	871	525	525
query94	686	399	295	295
query95	337	269	251	251
query96	488	616	278	278
query97	2710	2821	2717	2717
query98	225	207	193	193
query99	1690	1566	1453	1453
Total cold run time: 304460 ms
Total hot run time: 196422 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6061f3763916c27f0240d691ffc5a8b02210f680, data reload: false

query1	0.04	0.05	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.43	0.42	0.40
query6	1.15	0.65	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.51
query10	0.56	0.58	0.56
query11	0.16	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.60
query14	2.69	2.76	2.80
query15	0.88	0.82	0.82
query16	0.39	0.38	0.37
query17	1.08	0.99	1.05
query18	0.23	0.21	0.21
query19	1.97	1.86	1.96
query20	0.01	0.01	0.01
query21	15.37	0.89	0.56
query22	0.74	0.79	0.64
query23	15.34	1.40	0.58
query24	2.70	1.03	0.36
query25	0.12	0.16	0.16
query26	0.34	0.16	0.14
query27	0.05	0.06	0.04
query28	13.69	1.60	1.05
query29	12.59	3.88	3.24
query30	0.25	0.08	0.06
query31	2.84	0.58	0.37
query32	3.22	0.53	0.45
query33	3.07	3.15	3.07
query34	16.70	5.15	4.55
query35	4.58	4.51	4.47
query36	0.67	0.49	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.72 s
Total hot run time: 30.76 s

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32455 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 529978903c123793a431fd51678d445d110c9b5e, data reload: false

------ Round 1 ----------------------------------
q1	17560	6123	6005	6005
q2	2043	307	180	180
q3	10476	1245	737	737
q4	10208	874	446	446
q5	7505	2113	1954	1954
q6	203	188	150	150
q7	881	757	603	603
q8	9229	1330	1125	1125
q9	5225	4972	4886	4886
q10	6768	2304	1859	1859
q11	483	274	261	261
q12	340	353	225	225
q13	17786	3663	3018	3018
q14	219	234	218	218
q15	570	500	492	492
q16	647	621	592	592
q17	598	867	341	341
q18	7325	6660	6351	6351
q19	2061	965	547	547
q20	294	315	192	192
q21	2778	2252	1965	1965
q22	356	341	308	308
Total cold run time: 103555 ms
Total hot run time: 32455 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6265	6187	6228	6187
q2	233	326	233	233
q3	2235	2698	2326	2326
q4	1409	1846	1368	1368
q5	4378	4753	4750	4750
q6	186	180	147	147
q7	2042	1967	1782	1782
q8	2619	2842	2659	2659
q9	7314	7267	7240	7240
q10	3065	3367	2827	2827
q11	586	512	504	504
q12	640	745	606	606
q13	3343	3697	3171	3171
q14	286	296	296	296
q15	568	521	517	517
q16	641	704	682	682
q17	1216	1741	1254	1254
q18	7632	7390	7186	7186
q19	823	996	1158	996
q20	1971	2067	1963	1963
q21	5738	5334	4948	4948
q22	597	619	609	609
Total cold run time: 53787 ms
Total hot run time: 52251 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.90% (10123/26023)
Line Coverage: 29.90% (85559/286147)
Region Coverage: 29.01% (43703/150633)
Branch Coverage: 25.55% (22306/87308)
Coverage Report: http://coverage.selectdb-in.cc/coverage/529978903c123793a431fd51678d445d110c9b5e_529978903c123793a431fd51678d445d110c9b5e/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 195902 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 529978903c123793a431fd51678d445d110c9b5e, data reload: false

query1	1311	953	925	925
query2	6489	2340	2282	2282
query3	11102	4709	4704	4704
query4	33167	24127	23437	23437
query5	4623	620	460	460
query6	291	196	188	188
query7	3996	477	301	301
query8	322	244	238	238
query9	9684	2713	2701	2701
query10	453	316	250	250
query11	18211	15329	15149	15149
query12	158	109	103	103
query13	1582	552	406	406
query14	9988	7085	7046	7046
query15	244	229	200	200
query16	7853	674	476	476
query17	1546	751	595	595
query18	1856	409	346	346
query19	208	202	183	183
query20	117	129	126	126
query21	221	132	103	103
query22	4689	4510	4410	4410
query23	34536	33360	33499	33360
query24	6472	2317	2286	2286
query25	484	452	382	382
query26	971	276	155	155
query27	2480	461	333	333
query28	5830	2488	2455	2455
query29	671	555	427	427
query30	212	188	158	158
query31	995	917	855	855
query32	87	61	58	58
query33	511	348	292	292
query34	759	855	512	512
query35	806	836	760	760
query36	1023	1049	956	956
query37	117	93	76	76
query38	4430	4156	4216	4156
query39	1504	1472	1444	1444
query40	239	110	102	102
query41	46	44	42	42
query42	120	104	104	104
query43	543	539	508	508
query44	1286	798	831	798
query45	187	178	174	174
query46	893	1056	655	655
query47	2003	2022	1941	1941
query48	391	409	340	340
query49	717	454	389	389
query50	630	682	388	388
query51	7311	7421	7272	7272
query52	102	101	90	90
query53	227	251	190	190
query54	496	492	413	413
query55	83	82	79	79
query56	252	257	241	241
query57	1233	1242	1151	1151
query58	247	231	231	231
query59	3129	3456	3157	3157
query60	275	268	264	264
query61	109	107	123	107
query62	887	830	729	729
query63	230	204	184	184
query64	3766	1016	667	667
query65	3327	3226	3282	3226
query66	922	406	317	317
query67	16244	15878	15627	15627
query68	9993	760	523	523
query69	488	295	255	255
query70	1196	1155	1048	1048
query71	433	279	268	268
query72	6264	3948	3792	3792
query73	647	761	358	358
query74	10326	9215	8921	8921
query75	4615	3141	2639	2639
query76	5221	1211	784	784
query77	1024	349	281	281
query78	10116	10329	9382	9382
query79	5032	870	588	588
query80	671	519	421	421
query81	488	363	228	228
query82	236	148	123	123
query83	194	165	143	143
query84	281	89	74	74
query85	759	359	302	302
query86	342	283	309	283
query87	4757	4554	4334	4334
query88	3593	2233	2195	2195
query89	440	331	301	301
query90	2117	188	183	183
query91	130	132	106	106
query92	71	56	48	48
query93	3441	899	531	531
query94	673	360	302	302
query95	325	251	246	246
query96	489	598	278	278
query97	2756	2797	2629	2629
query98	221	201	200	200
query99	1691	1541	1456	1456
Total cold run time: 304924 ms
Total hot run time: 195902 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 529978903c123793a431fd51678d445d110c9b5e, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.08	0.07
query4	1.60	0.10	0.11
query5	0.40	0.41	0.41
query6	1.17	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.51	0.52
query10	0.55	0.56	0.55
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.62	0.59
query14	2.74	2.76	2.82
query15	0.89	0.83	0.83
query16	0.38	0.37	0.37
query17	1.00	1.02	1.01
query18	0.23	0.21	0.20
query19	1.96	1.84	2.05
query20	0.02	0.01	0.01
query21	15.74	0.96	0.60
query22	1.01	0.79	0.76
query23	15.18	1.45	0.54
query24	3.02	1.71	0.85
query25	0.21	0.22	0.06
query26	0.19	0.16	0.13
query27	0.05	0.05	0.05
query28	13.84	1.49	1.03
query29	12.59	3.99	3.30
query30	0.25	0.09	0.06
query31	2.81	0.60	0.40
query32	3.22	0.55	0.45
query33	3.22	3.15	3.07
query34	16.66	5.13	4.51
query35	4.57	4.51	4.49
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.55 s
Total hot run time: 31.28 s

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Contributor

github-actions bot commented Jan 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants