Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](ES Catalog)Make sure ES meta is synced before using #46781

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[fix](ES Catalog)Make sure ES meta is synced before using #46781

wants to merge 4 commits into from
+66 −119

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Jan 10, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #46780

Problem Summary:

  1. Added calls to EsResource.fillUrlsWithSchema and syncTableMetaData to ensure ES table URLs are correctly parsed and metadata is synchronized before use
  2. Introduced executeWithRetry function to handle retries for ES queries that may fail due to unsynchronized metadata.
  3. Remove useless ES tests and config.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@qidaye
Copy link
Contributor Author

qidaye commented Jan 10, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32775 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1a6bb9b5b271e079eb4d623351dc1913a4b5592, data reload: false

------ Round 1 ----------------------------------
q1	17556	6130	6085	6085
q2	2055	307	182	182
q3	10557	1232	742	742
q4	10222	881	437	437
q5	8045	2179	1950	1950
q6	211	181	149	149
q7	894	741	603	603
q8	9245	1340	1104	1104
q9	5429	5124	5008	5008
q10	6781	2287	1846	1846
q11	479	284	265	265
q12	343	362	220	220
q13	18470	3647	3057	3057
q14	228	239	211	211
q15	566	513	507	507
q16	651	617	595	595
q17	589	849	346	346
q18	6946	6482	6402	6402
q19	2952	949	525	525
q20	298	328	189	189
q21	2853	2266	2045	2045
q22	370	332	307	307
Total cold run time: 105740 ms
Total hot run time: 32775 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6444	6252	6267	6252
q2	238	324	233	233
q3	2272	2678	2339	2339
q4	1411	1771	1444	1444
q5	4311	4771	4902	4771
q6	185	182	143	143
q7	2079	1950	1815	1815
q8	2614	2852	2707	2707
q9	7311	7304	7399	7304
q10	3049	3241	2765	2765
q11	668	528	509	509
q12	699	764	617	617
q13	3475	3859	3269	3269
q14	273	299	272	272
q15	546	510	496	496
q16	676	688	681	681
q17	1221	1731	1273	1273
q18	7796	7534	7296	7296
q19	822	1129	1113	1113
q20	1957	2041	1862	1862
q21	5685	5276	5151	5151
q22	640	621	556	556
Total cold run time: 54372 ms
Total hot run time: 52868 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194411 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1a6bb9b5b271e079eb4d623351dc1913a4b5592, data reload: false

query1	1341	970	923	923
query2	6340	2459	2288	2288
query3	10978	4668	4895	4668
query4	32822	23612	23255	23255
query5	4094	605	474	474
query6	289	196	190	190
query7	3993	496	316	316
query8	303	246	249	246
query9	9294	2762	2762	2762
query10	459	320	261	261
query11	17572	15272	15083	15083
query12	165	108	105	105
query13	1559	527	393	393
query14	9999	6591	7501	6591
query15	268	203	188	188
query16	8319	674	484	484
query17	1540	786	588	588
query18	2075	429	310	310
query19	220	179	152	152
query20	122	124	117	117
query21	212	127	108	108
query22	4727	4619	4160	4160
query23	34498	33461	33284	33284
query24	6456	2428	2304	2304
query25	476	461	390	390
query26	1147	279	158	158
query27	2482	470	336	336
query28	5910	2517	2504	2504
query29	632	552	426	426
query30	212	191	157	157
query31	1031	888	800	800
query32	71	61	56	56
query33	485	346	301	301
query34	793	849	528	528
query35	794	818	748	748
query36	1007	1004	954	954
query37	133	99	78	78
query38	4389	4134	4105	4105
query39	1512	1504	1470	1470
query40	207	117	105	105
query41	51	51	49	49
query42	122	101	102	101
query43	515	538	493	493
query44	1307	842	855	842
query45	193	177	172	172
query46	902	1063	656	656
query47	1875	1901	1852	1852
query48	389	405	335	335
query49	769	487	389	389
query50	656	649	395	395
query51	7110	7110	6952	6952
query52	103	105	89	89
query53	229	251	193	193
query54	486	498	415	415
query55	86	78	81	78
query56	267	259	264	259
query57	1213	1207	1150	1150
query58	251	228	236	228
query59	3392	3428	3160	3160
query60	283	264	252	252
query61	117	111	111	111
query62	888	805	730	730
query63	229	195	187	187
query64	3786	1002	664	664
query65	3370	3219	3234	3219
query66	945	411	298	298
query67	16043	16161	15347	15347
query68	7798	692	520	520
query69	455	287	247	247
query70	1246	1090	1126	1090
query71	436	286	246	246
query72	6194	3911	3881	3881
query73	649	772	357	357
query74	10041	9044	8867	8867
query75	4039	3169	2683	2683
query76	3622	1180	775	775
query77	758	375	273	273
query78	9924	10010	9322	9322
query79	3473	805	602	602
query80	622	513	537	513
query81	489	281	247	247
query82	657	150	124	124
query83	205	172	149	149
query84	283	91	78	78
query85	758	344	289	289
query86	347	307	303	303
query87	4516	4371	4284	4284
query88	4505	2204	2224	2204
query89	422	322	294	294
query90	1830	190	194	190
query91	133	139	104	104
query92	67	56	52	52
query93	2004	875	528	528
query94	637	398	288	288
query95	332	262	259	259
query96	497	607	283	283
query97	2864	2946	2807	2807
query98	228	201	200	200
query99	1610	1524	1370	1370
Total cold run time: 296144 ms
Total hot run time: 194411 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1a6bb9b5b271e079eb4d623351dc1913a4b5592, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.40	0.41	0.43
query6	1.15	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.59	0.50	0.52
query10	0.56	0.56	0.54
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.84	2.76	2.84
query15	0.89	0.83	0.83
query16	0.39	0.38	0.40
query17	1.08	1.05	1.06
query18	0.23	0.21	0.21
query19	1.97	1.90	1.99
query20	0.01	0.01	0.01
query21	15.35	0.98	0.60
query22	0.74	0.74	0.75
query23	15.25	1.49	0.55
query24	2.72	1.59	1.82
query25	0.28	0.17	0.20
query26	0.26	0.15	0.13
query27	0.06	0.05	0.04
query28	14.11	1.48	1.04
query29	12.60	3.90	3.32
query30	0.24	0.09	0.06
query31	2.83	0.60	0.38
query32	3.24	0.54	0.47
query33	3.13	3.06	3.10
query34	16.79	5.10	4.42
query35	4.49	4.43	4.50
query36	0.65	0.49	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.23 s
Total hot run time: 32.08 s

@qidaye qidaye force-pushed the fix_es_query_null branch from e1a6bb9 to 8dfca62 Compare January 10, 2025 12:03
@qidaye
Copy link
Contributor Author

qidaye commented Jan 10, 2025

run buildall

@qidaye qidaye force-pushed the fix_es_query_null branch from 8dfca62 to 94ff990 Compare January 10, 2025 12:17
@qidaye
Copy link
Contributor Author

qidaye commented Jan 10, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32439 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94ff9901a682fa99a58953dfd48b8832bdbb734c, data reload: false

------ Round 1 ----------------------------------
q1	17606	6165	6032	6032
q2	2086	308	183	183
q3	10916	1228	747	747
q4	10302	877	448	448
q5	7509	2161	1932	1932
q6	199	177	148	148
q7	881	787	599	599
q8	9234	1347	1137	1137
q9	5185	4888	4917	4888
q10	6753	2295	1882	1882
q11	489	276	270	270
q12	333	363	212	212
q13	17775	3665	3047	3047
q14	227	244	212	212
q15	572	507	503	503
q16	637	636	595	595
q17	555	849	323	323
q18	7059	6445	6276	6276
q19	1556	945	536	536
q20	296	317	192	192
q21	2853	2202	1976	1976
q22	362	332	301	301
Total cold run time: 103385 ms
Total hot run time: 32439 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6234	6209	6192	6192
q2	236	329	237	237
q3	2232	2630	2338	2338
q4	1391	1838	1338	1338
q5	4357	4755	4777	4755
q6	178	174	141	141
q7	2072	1995	1791	1791
q8	2585	2795	2695	2695
q9	7274	7255	7328	7255
q10	3070	3358	2749	2749
q11	573	508	489	489
q12	694	796	610	610
q13	3547	3850	3283	3283
q14	313	322	293	293
q15	566	517	500	500
q16	647	719	634	634
q17	1204	1738	1260	1260
q18	7742	7471	7446	7446
q19	776	1067	1098	1067
q20	1965	2023	1979	1979
q21	5694	5078	4918	4918
q22	604	616	582	582
Total cold run time: 53954 ms
Total hot run time: 52552 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195126 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94ff9901a682fa99a58953dfd48b8832bdbb734c, data reload: false

query1	1323	948	944	944
query2	6331	2386	2340	2340
query3	11003	4779	4815	4779
query4	33107	23594	23245	23245
query5	3655	610	468	468
query6	284	193	202	193
query7	3992	491	317	317
query8	301	232	225	225
query9	9271	2646	2659	2646
query10	457	308	239	239
query11	17962	15306	15044	15044
query12	163	116	109	109
query13	1556	503	380	380
query14	9972	7250	7219	7219
query15	237	206	203	203
query16	8058	632	471	471
query17	1559	766	565	565
query18	2094	411	311	311
query19	211	206	188	188
query20	119	118	111	111
query21	200	122	103	103
query22	4573	4538	4375	4375
query23	34185	33372	33212	33212
query24	6543	2321	2335	2321
query25	481	475	392	392
query26	867	285	161	161
query27	1993	470	323	323
query28	5399	2447	2416	2416
query29	626	584	418	418
query30	210	189	157	157
query31	960	928	810	810
query32	80	63	55	55
query33	538	372	312	312
query34	761	870	522	522
query35	793	832	735	735
query36	1017	1035	959	959
query37	112	98	71	71
query38	4220	4184	4082	4082
query39	1467	1460	1405	1405
query40	222	124	106	106
query41	52	56	48	48
query42	153	107	103	103
query43	523	543	503	503
query44	1309	806	809	806
query45	179	175	174	174
query46	860	1045	653	653
query47	1883	1942	1841	1841
query48	382	389	329	329
query49	724	518	404	404
query50	658	664	405	405
query51	7053	7112	6831	6831
query52	124	100	88	88
query53	219	253	182	182
query54	478	514	408	408
query55	84	82	75	75
query56	253	282	244	244
query57	1229	1176	1096	1096
query58	237	237	245	237
query59	3161	3203	3270	3203
query60	275	284	260	260
query61	111	111	114	111
query62	864	795	757	757
query63	240	198	197	197
query64	3512	1132	746	746
query65	3397	3232	3249	3232
query66	853	440	410	410
query67	16376	15774	15544	15544
query68	8299	699	510	510
query69	487	289	256	256
query70	1206	1101	1131	1101
query71	431	292	248	248
query72	6471	3872	3822	3822
query73	656	745	362	362
query74	10241	9049	8690	8690
query75	3931	3148	2671	2671
query76	3635	1190	749	749
query77	758	373	282	282
query78	9915	9986	9377	9377
query79	3582	786	587	587
query80	728	521	432	432
query81	483	288	230	230
query82	591	166	132	132
query83	196	167	172	167
query84	276	102	81	81
query85	759	350	354	350
query86	345	315	290	290
query87	4532	4446	4543	4446
query88	4046	2185	2129	2129
query89	408	320	292	292
query90	1910	190	193	190
query91	129	139	106	106
query92	66	58	49	49
query93	1388	874	539	539
query94	634	378	279	279
query95	325	259	263	259
query96	478	603	282	282
query97	2873	2943	2808	2808
query98	235	223	195	195
query99	1636	1489	1396	1396
Total cold run time: 293741 ms
Total hot run time: 195126 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94ff9901a682fa99a58953dfd48b8832bdbb734c, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.10
query5	0.43	0.42	0.42
query6	1.15	0.65	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.51
query10	0.54	0.56	0.54
query11	0.13	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	2.72	2.73	2.71
query15	0.90	0.82	0.83
query16	0.40	0.40	0.39
query17	0.98	1.08	0.99
query18	0.23	0.20	0.20
query19	1.99	1.83	2.02
query20	0.02	0.02	0.01
query21	15.35	0.91	0.58
query22	0.75	0.77	0.63
query23	15.39	1.47	0.65
query24	3.49	0.50	1.17
query25	0.19	0.14	0.09
query26	0.36	0.14	0.13
query27	0.07	0.06	0.06
query28	13.95	1.56	1.04
query29	12.59	3.94	3.24
query30	0.25	0.09	0.07
query31	2.82	0.62	0.39
query32	3.22	0.54	0.46
query33	3.03	3.19	3.11
query34	16.52	5.15	4.47
query35	4.52	4.46	4.53
query36	0.68	0.51	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.53 s
Total hot run time: 30.8 s

@qidaye
Copy link
Contributor Author

qidaye commented Jan 13, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11a68810b9adc731aa1406e8b6c487e1bf21bf6a, data reload: false

------ Round 1 ----------------------------------
q1	17577	6126	6059	6059
q2	2061	300	175	175
q3	10413	1238	698	698
q4	10215	857	442	442
q5	7529	2173	1946	1946
q6	203	176	147	147
q7	876	737	600	600
q8	9252	1362	1171	1171
q9	5113	4876	4812	4812
q10	6745	2294	1848	1848
q11	491	282	254	254
q12	341	358	218	218
q13	17783	3652	3086	3086
q14	234	252	211	211
q15	556	502	502	502
q16	621	613	599	599
q17	577	852	328	328
q18	6977	6504	6300	6300
q19	1207	972	548	548
q20	319	323	191	191
q21	2811	2173	1996	1996
q22	368	335	332	332
Total cold run time: 102269 ms
Total hot run time: 32463 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6215	6195	6195	6195
q2	248	337	239	239
q3	2258	2588	2296	2296
q4	1393	1808	1350	1350
q5	4328	4689	4605	4605
q6	182	179	146	146
q7	2061	1951	1842	1842
q8	2628	2808	2708	2708
q9	7226	7166	7136	7136
q10	2994	3301	2806	2806
q11	579	537	505	505
q12	673	770	580	580
q13	3460	3816	3243	3243
q14	282	297	303	297
q15	557	504	512	504
q16	647	690	646	646
q17	1201	1717	1240	1240
q18	7627	7447	7089	7089
q19	765	1092	1023	1023
q20	1915	1984	1855	1855
q21	5397	5176	4782	4782
q22	602	609	583	583
Total cold run time: 53238 ms
Total hot run time: 51670 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11a68810b9adc731aa1406e8b6c487e1bf21bf6a, data reload: false

query1	989	369	373	369
query2	6535	2388	2332	2332
query3	6708	212	209	209
query4	34212	23250	23207	23207
query5	4368	605	470	470
query6	311	209	180	180
query7	4623	499	294	294
query8	299	245	223	223
query9	9701	2709	2662	2662
query10	481	312	254	254
query11	17735	15236	15109	15109
query12	160	106	106	106
query13	1658	536	412	412
query14	9990	7415	7075	7075
query15	257	204	183	183
query16	8856	602	435	435
query17	1629	729	550	550
query18	2107	403	303	303
query19	204	177	177	177
query20	123	114	113	113
query21	212	122	101	101
query22	4201	4176	4218	4176
query23	33847	33012	32871	32871
query24	6341	2260	2237	2237
query25	503	446	386	386
query26	1174	264	153	153
query27	2001	452	351	351
query28	5313	2426	2397	2397
query29	723	543	408	408
query30	229	194	164	164
query31	975	820	790	790
query32	93	63	60	60
query33	509	352	300	300
query34	754	841	494	494
query35	805	802	742	742
query36	1036	1029	937	937
query37	115	93	78	78
query38	3956	4075	4059	4059
query39	1457	1483	1396	1396
query40	208	115	102	102
query41	53	51	52	51
query42	123	105	105	105
query43	511	500	491	491
query44	1327	805	800	800
query45	181	169	162	162
query46	842	1072	638	638
query47	1843	1851	1789	1789
query48	391	387	312	312
query49	783	475	398	398
query50	631	643	396	396
query51	7011	6960	6879	6879
query52	104	110	91	91
query53	220	249	178	178
query54	472	499	403	403
query55	81	80	76	76
query56	271	252	237	237
query57	1196	1185	1062	1062
query58	250	231	243	231
query59	3090	3001	3013	3001
query60	296	267	318	267
query61	116	119	132	119
query62	859	758	698	698
query63	223	186	189	186
query64	4117	989	683	683
query65	3290	3151	3160	3151
query66	1061	422	316	316
query67	15930	15583	15378	15378
query68	7728	687	513	513
query69	476	295	263	263
query70	1200	1094	1093	1093
query71	446	289	254	254
query72	6500	3887	3751	3751
query73	654	740	360	360
query74	10129	8873	8749	8749
query75	3749	3135	2645	2645
query76	3706	1182	750	750
query77	776	353	288	288
query78	9968	9957	9413	9413
query79	3382	783	570	570
query80	723	526	428	428
query81	480	283	241	241
query82	598	149	135	135
query83	204	177	151	151
query84	286	91	81	81
query85	772	358	307	307
query86	355	393	311	311
query87	4534	4299	4351	4299
query88	3917	2172	2122	2122
query89	396	334	285	285
query90	1891	193	195	193
query91	134	135	107	107
query92	70	64	50	50
query93	992	879	537	537
query94	670	412	301	301
query95	342	273	247	247
query96	483	607	276	276
query97	2905	2951	2809	2809
query98	217	205	196	196
query99	1641	1462	1376	1376
Total cold run time: 290994 ms
Total hot run time: 188064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11a68810b9adc731aa1406e8b6c487e1bf21bf6a, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.66	0.11	0.10
query5	0.41	0.39	0.42
query6	1.17	0.66	0.67
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.49
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.60	0.60
query14	2.82	2.86	2.72
query15	0.89	0.84	0.83
query16	0.38	0.37	0.39
query17	1.02	0.99	1.05
query18	0.23	0.21	0.20
query19	1.94	1.87	1.98
query20	0.01	0.01	0.01
query21	15.36	0.97	0.60
query22	0.77	0.81	0.59
query23	15.29	1.45	0.54
query24	3.08	1.78	1.57
query25	0.23	0.09	0.12
query26	0.27	0.13	0.14
query27	0.08	0.05	0.04
query28	14.53	1.47	1.04
query29	12.56	4.09	3.42
query30	0.26	0.09	0.06
query31	2.82	0.60	0.38
query32	3.24	0.55	0.46
query33	3.14	3.22	3.16
query34	16.62	5.12	4.47
query35	4.46	4.46	4.48
query36	0.65	0.48	0.47
query37	0.09	0.07	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.8 s
Total hot run time: 31.96 s

@qidaye
Copy link
Contributor Author

qidaye commented Jan 14, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32645 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8317e26b204b476aca32b30da902bfb403b84c3c, data reload: false

------ Round 1 ----------------------------------
q1	17579	6179	6032	6032
q2	2054	323	180	180
q3	10383	1237	740	740
q4	10221	857	429	429
q5	7615	2195	1911	1911
q6	207	177	150	150
q7	891	759	613	613
q8	9246	1357	1130	1130
q9	5142	4937	4855	4855
q10	6799	2320	1846	1846
q11	475	285	257	257
q12	341	364	224	224
q13	17788	3657	3068	3068
q14	245	231	212	212
q15	600	544	519	519
q16	638	621	607	607
q17	566	849	326	326
q18	7012	6540	6467	6467
q19	2770	963	564	564
q20	300	321	190	190
q21	2852	2101	2017	2017
q22	375	330	308	308
Total cold run time: 104099 ms
Total hot run time: 32645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6371	6208	6185	6185
q2	235	336	238	238
q3	2304	2637	2342	2342
q4	1401	1797	1360	1360
q5	4317	4714	4784	4714
q6	184	179	144	144
q7	2061	2012	1799	1799
q8	2616	2834	2693	2693
q9	7275	7249	7357	7249
q10	3077	3301	2732	2732
q11	578	515	487	487
q12	693	839	621	621
q13	3556	3900	3307	3307
q14	300	301	278	278
q15	576	535	515	515
q16	630	701	672	672
q17	1229	1743	1284	1284
q18	7719	7589	7421	7421
q19	812	1154	1033	1033
q20	1998	2059	1916	1916
q21	5666	5065	5001	5001
q22	643	601	605	601
Total cold run time: 54241 ms
Total hot run time: 52592 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195582 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8317e26b204b476aca32b30da902bfb403b84c3c, data reload: false

query1	1306	980	932	932
query2	6208	2165	2132	2132
query3	10977	4716	4548	4548
query4	61867	28923	23387	23387
query5	5504	618	471	471
query6	422	213	218	213
query7	5533	507	300	300
query8	336	230	227	227
query9	8039	2679	2663	2663
query10	452	332	263	263
query11	17517	15269	15583	15269
query12	167	115	108	108
query13	1391	541	446	446
query14	11682	6665	6856	6665
query15	235	216	186	186
query16	7256	581	475	475
query17	1103	709	568	568
query18	1897	409	316	316
query19	207	180	161	161
query20	129	115	110	110
query21	218	123	104	104
query22	4630	4697	4407	4407
query23	34350	34263	33582	33582
query24	5592	2336	2359	2336
query25	471	446	392	392
query26	665	265	167	167
query27	1671	487	347	347
query28	4144	2507	2474	2474
query29	583	591	465	465
query30	210	197	167	167
query31	927	910	840	840
query32	71	65	62	62
query33	444	373	338	338
query34	778	895	521	521
query35	816	834	749	749
query36	1033	1058	966	966
query37	133	110	81	81
query38	4270	4404	4383	4383
query39	1482	1445	1487	1445
query40	209	121	104	104
query41	51	49	50	49
query42	121	105	104	104
query43	532	546	514	514
query44	1344	849	848	848
query45	187	180	171	171
query46	928	1077	669	669
query47	1914	1931	1817	1817
query48	391	427	312	312
query49	734	500	434	434
query50	654	666	400	400
query51	7000	7066	6910	6910
query52	108	105	94	94
query53	233	256	193	193
query54	491	525	418	418
query55	82	86	74	74
query56	268	270	263	263
query57	1218	1209	1135	1135
query58	235	241	241	241
query59	3342	3298	3050	3050
query60	273	272	274	272
query61	117	109	113	109
query62	816	789	728	728
query63	223	195	195	195
query64	1284	1046	687	687
query65	3269	3156	3153	3153
query66	646	413	311	311
query67	15780	15554	15357	15357
query68	4763	854	552	552
query69	490	290	267	267
query70	1245	1129	1116	1116
query71	405	284	258	258
query72	6103	3868	3990	3868
query73	752	751	380	380
query74	10240	9212	8861	8861
query75	3244	3194	2696	2696
query76	3521	1231	757	757
query77	510	389	344	344
query78	10302	10053	9331	9331
query79	3356	839	597	597
query80	1644	542	448	448
query81	569	290	248	248
query82	449	155	127	127
query83	267	177	154	154
query84	288	103	87	87
query85	807	364	310	310
query86	454	331	317	317
query87	4623	4463	4420	4420
query88	4814	2226	2237	2226
query89	414	334	292	292
query90	1606	192	192	192
query91	136	134	108	108
query92	72	58	61	58
query93	2862	897	538	538
query94	766	387	299	299
query95	329	264	257	257
query96	499	648	279	279
query97	2873	2870	2742	2742
query98	231	202	211	202
query99	1386	1514	1373	1373
Total cold run time: 317186 ms
Total hot run time: 195582 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8317e26b204b476aca32b30da902bfb403b84c3c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.11
query5	0.42	0.43	0.42
query6	1.15	0.66	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.58	0.50	0.50
query10	0.57	0.56	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.84	2.73	2.72
query15	0.90	0.85	0.83
query16	0.38	0.37	0.39
query17	1.04	1.06	1.05
query18	0.24	0.21	0.20
query19	1.99	1.87	1.95
query20	0.02	0.02	0.01
query21	15.37	0.91	0.59
query22	0.77	0.75	0.66
query23	15.35	1.48	0.65
query24	2.87	1.11	1.45
query25	0.12	0.19	0.07
query26	0.24	0.14	0.14
query27	0.06	0.07	0.05
query28	14.13	1.56	1.04
query29	12.57	4.03	3.30
query30	0.26	0.09	0.07
query31	2.81	0.60	0.39
query32	3.23	0.56	0.47
query33	3.04	3.01	3.04
query34	16.57	5.23	4.50
query35	4.53	4.52	4.50
query36	0.64	0.50	0.49
query37	0.10	0.06	0.05
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.1 s
Total hot run time: 31.59 s

@qidaye
Copy link
Contributor Author

qidaye commented Jan 14, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33006 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccec04f4fa0ca463e2365837dafa80bc09a849c4, data reload: false

------ Round 1 ----------------------------------
q1	17596	6199	6064	6064
q2	2049	306	180	180
q3	10405	1267	716	716
q4	10214	871	455	455
q5	7550	2224	2003	2003
q6	210	182	152	152
q7	896	754	613	613
q8	9253	1406	1188	1188
q9	5420	4914	4910	4910
q10	6755	2318	1850	1850
q11	486	276	258	258
q12	346	357	226	226
q13	17761	3734	3110	3110
q14	227	247	218	218
q15	571	542	520	520
q16	638	601	583	583
q17	565	863	319	319
q18	6999	6613	6560	6560
q19	1223	958	551	551
q20	314	342	200	200
q21	2833	2193	2018	2018
q22	376	331	312	312
Total cold run time: 102687 ms
Total hot run time: 33006 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6228	6231	6243	6231
q2	248	339	247	247
q3	2290	2690	2317	2317
q4	1361	1837	1375	1375
q5	4354	4756	4815	4756
q6	195	186	148	148
q7	2043	2003	1895	1895
q8	2613	2801	2683	2683
q9	7340	7217	7164	7164
q10	3024	3331	2807	2807
q11	597	527	496	496
q12	678	782	654	654
q13	3419	3927	3255	3255
q14	279	299	266	266
q15	590	525	527	525
q16	654	698	682	682
q17	1212	1748	1260	1260
q18	7801	7529	7456	7456
q19	798	910	1148	910
q20	1972	2068	1904	1904
q21	5601	5319	5016	5016
q22	599	593	618	593
Total cold run time: 53896 ms
Total hot run time: 52640 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188243 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccec04f4fa0ca463e2365837dafa80bc09a849c4, data reload: false

query1	950	381	381	381
query2	6529	2112	2069	2069
query3	6708	218	230	218
query4	33499	23448	23712	23448
query5	4323	648	494	494
query6	297	209	200	200
query7	4620	492	309	309
query8	307	250	234	234
query9	9554	2752	2740	2740
query10	469	309	253	253
query11	18061	15456	15011	15011
query12	162	110	110	110
query13	1654	530	422	422
query14	9455	6545	7748	6545
query15	255	188	197	188
query16	8540	586	470	470
query17	1646	720	551	551
query18	2102	386	302	302
query19	221	184	150	150
query20	117	142	118	118
query21	207	121	101	101
query22	4283	4356	4073	4073
query23	34308	33607	33325	33325
query24	6543	2201	2352	2201
query25	510	446	385	385
query26	1196	275	157	157
query27	1988	463	348	348
query28	5131	2462	2426	2426
query29	706	541	437	437
query30	238	191	157	157
query31	974	876	811	811
query32	89	64	60	60
query33	514	345	309	309
query34	754	853	505	505
query35	782	812	728	728
query36	1016	1037	905	905
query37	121	103	101	101
query38	4149	4134	4043	4043
query39	1476	1390	1385	1385
query40	202	121	101	101
query41	54	53	53	53
query42	121	99	105	99
query43	519	530	488	488
query44	1357	830	848	830
query45	187	172	169	169
query46	867	1065	654	654
query47	1799	1814	1745	1745
query48	394	407	313	313
query49	761	497	394	394
query50	623	659	404	404
query51	6921	6872	6938	6872
query52	105	110	94	94
query53	229	257	190	190
query54	485	524	431	431
query55	84	86	85	85
query56	293	274	265	265
query57	1160	1162	1066	1066
query58	243	236	240	236
query59	2984	3000	2876	2876
query60	291	279	269	269
query61	132	124	122	122
query62	824	762	703	703
query63	229	189	193	189
query64	4049	1021	664	664
query65	3325	3161	3193	3161
query66	818	425	332	332
query67	15743	15400	15448	15400
query68	2305	825	560	560
query69	457	298	261	261
query70	1187	1164	1143	1143
query71	312	289	260	260
query72	5728	3836	3881	3836
query73	644	754	369	369
query74	9884	9192	8749	8749
query75	3140	3206	2683	2683
query76	2192	1198	780	780
query77	436	367	286	286
query78	9926	10065	9295	9295
query79	3021	811	606	606
query80	1638	532	453	453
query81	563	282	237	237
query82	359	153	122	122
query83	266	180	158	158
query84	242	98	78	78
query85	789	344	304	304
query86	466	316	310	310
query87	4457	4434	4298	4298
query88	4205	2206	2188	2188
query89	390	332	294	294
query90	1880	193	191	191
query91	135	136	107	107
query92	63	63	56	56
query93	1786	882	546	546
query94	704	402	293	293
query95	343	273	256	256
query96	488	605	282	282
query97	2810	2894	2795	2795
query98	226	202	195	195
query99	1416	1508	1377	1377
Total cold run time: 281711 ms
Total hot run time: 188243 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccec04f4fa0ca463e2365837dafa80bc09a849c4, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.42
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.51	0.51
query10	0.55	0.57	0.55
query11	0.15	0.12	0.11
query12	0.14	0.10	0.11
query13	0.61	0.61	0.62
query14	2.85	2.83	2.77
query15	0.90	0.82	0.82
query16	0.38	0.39	0.37
query17	1.04	1.04	1.06
query18	0.23	0.21	0.21
query19	1.94	1.77	1.98
query20	0.02	0.01	0.02
query21	15.39	0.93	0.58
query22	0.75	0.82	0.59
query23	15.37	1.43	0.63
query24	2.62	0.66	2.18
query25	0.21	0.16	0.16
query26	0.24	0.16	0.13
query27	0.06	0.06	0.04
query28	14.20	1.51	1.04
query29	12.67	3.99	3.28
query30	0.26	0.09	0.06
query31	2.83	0.61	0.37
query32	3.23	0.55	0.47
query33	2.98	3.01	3.09
query34	16.73	5.22	4.55
query35	4.53	4.52	4.59
query36	0.64	0.51	0.48
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.18 s
Total hot run time: 31.08 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dev/2.1.x dev/3.0.x
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] test_es_query failed flaky
4 participants
@qidaye @Thearas @doris-robot @morningman