Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

branch-3.0: [opt](load) Add config to control commit lock scope for tables #46996 #47039

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from
Open

branch-3.0: [opt](load) Add config to control commit lock scope for tables #46996 #47039

wants to merge 1 commit into from
+31 −13

Conversation

liaoxin01
Copy link
Contributor

cherry pick from #46996

@liaoxin01
Copy link
Contributor Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
[opt](load) Add config to control commit lock scope for tables (apach…
fbb5045 
…e#46996)

Problem Summary:
Previously, all tables required commit locks during transaction commit,
which helped reduce conflicts at the MetaService level. However, this
approach may not be optimal for all scenarios since only MOW
(Merge-on-Write) tables truly need strict concurrency control.

This PR adds a new config `enable_commit_lock_for_all_tables` (default:
true) to control the commit lock strategy:

- When enabled (default): All tables will acquire commit locks during
transaction commit, which helps reduce conflicts at MetaService level by
queueing transactions at FE level
- When disabled: Only MOW tables will acquire commit locks, which may
improve concurrency for non-MOW tables but could increase conflicts at
MetaService level

The default setting maintains the original behavior to avoid potential
performance impact from increased MetaService conflicts, while providing
flexibility to optimize for different deployment scenarios.
@liaoxin01 liaoxin01 force-pushed the pick_46996_to_origin_branch-3.0 branch from 866f590 to fbb5045 Compare January 15, 2025 14:22
@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41686 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fbb5045abbdeb572993078b757fb9d81487d0c0e, data reload: false

------ Round 1 ----------------------------------
q1	17655	7550	7418	7418
q2	2055	174	183	174
q3	10599	1151	1203	1151
q4	10504	792	820	792
q5	7895	2904	2937	2904
q6	239	149	149	149
q7	998	628	620	620
q8	9918	2036	2052	2036
q9	6863	6482	6493	6482
q10	9366	2375	2354	2354
q11	482	292	271	271
q12	829	230	220	220
q13	18596	3050	3037	3037
q14	235	217	220	217
q15	565	537	531	531
q16	704	635	608	608
q17	997	634	625	625
q18	7337	6796	6567	6567
q19	1392	1099	976	976
q20	506	207	205	205
q21	4000	3361	3343	3343
q22	1075	1006	1008	1006
Total cold run time: 112810 ms
Total hot run time: 41686 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7293	7184	7286	7184
q2	343	237	236	236
q3	3230	3031	2987	2987
q4	2074	1854	1777	1777
q5	5767	5758	5801	5758
q6	226	143	144	143
q7	2184	1799	1845	1799
q8	3317	3558	3508	3508
q9	8962	8834	8981	8834
q10	3593	3594	3570	3570
q11	613	508	492	492
q12	812	602	592	592
q13	8246	3112	3176	3112
q14	313	263	273	263
q15	576	528	517	517
q16	703	674	668	668
q17	1854	1630	1577	1577
q18	8323	7711	7543	7543
q19	1686	1626	1444	1444
q20	2059	1818	1814	1814
q21	5359	5177	5047	5047
q22	1093	998	1008	998
Total cold run time: 68626 ms
Total hot run time: 59863 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192794 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fbb5045abbdeb572993078b757fb9d81487d0c0e, data reload: false

query1	1016	397	367	367
query2	6506	2135	2088	2088
query3	6713	218	216	216
query4	34073	23541	23466	23466
query5	4303	457	444	444
query6	273	171	170	170
query7	4620	306	315	306
query8	285	231	219	219
query9	9693	2669	2656	2656
query10	476	267	248	248
query11	18252	15243	15246	15243
query12	156	104	107	104
query13	1649	427	423	423
query14	10399	7562	7587	7562
query15	290	192	186	186
query16	8198	429	475	429
query17	1715	578	562	562
query18	2155	302	317	302
query19	371	161	160	160
query20	117	108	106	106
query21	207	104	107	104
query22	4385	4382	4375	4375
query23	35119	34023	34015	34015
query24	11302	2867	2901	2867
query25	675	431	422	422
query26	1496	171	175	171
query27	2867	347	350	347
query28	7887	2429	2477	2429
query29	900	454	447	447
query30	322	164	171	164
query31	1047	813	811	811
query32	95	60	60	60
query33	782	310	299	299
query34	936	512	528	512
query35	880	715	726	715
query36	1106	924	949	924
query37	130	76	72	72
query38	3973	3864	3857	3857
query39	1469	1424	1448	1424
query40	291	106	103	103
query41	53	50	51	50
query42	115	104	103	103
query43	556	493	496	493
query44	1257	796	800	796
query45	186	171	167	167
query46	1151	726	723	723
query47	1922	1810	1827	1810
query48	471	381	423	381
query49	1267	409	410	409
query50	815	408	421	408
query51	7392	7202	7212	7202
query52	102	94	95	94
query53	262	185	190	185
query54	1183	472	466	466
query55	81	82	81	81
query56	288	261	266	261
query57	1238	1125	1122	1122
query58	249	218	221	218
query59	3268	2891	2945	2891
query60	286	258	253	253
query61	110	110	110	110
query62	893	678	677	677
query63	221	197	188	188
query64	5172	667	654	654
query65	3358	3224	3228	3224
query66	1458	312	349	312
query67	16035	15383	15765	15383
query68	4780	565	558	558
query69	452	284	271	271
query70	1120	1100	1118	1100
query71	418	267	269	267
query72	6491	4205	4052	4052
query73	780	341	342	341
query74	10433	9198	9098	9098
query75	3421	2681	2661	2661
query76	2955	1111	1118	1111
query77	422	281	271	271
query78	10319	9528	9627	9528
query79	1690	582	600	582
query80	1035	452	427	427
query81	550	244	248	244
query82	943	123	118	118
query83	217	149	145	145
query84	235	82	76	76
query85	1276	309	290	290
query86	373	271	280	271
query87	4595	4296	4272	4272
query88	3537	2406	2348	2348
query89	403	293	295	293
query90	1928	191	185	185
query91	182	154	148	148
query92	60	52	52	52
query93	1074	556	548	548
query94	877	289	281	281
query95	352	258	258	258
query96	604	285	278	278
query97	3328	3186	3220	3186
query98	220	201	195	195
query99	1544	1337	1303	1303
Total cold run time: 303132 ms
Total hot run time: 192794 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fbb5045abbdeb572993078b757fb9d81487d0c0e, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.07
query4	1.63	0.10	0.10
query5	0.52	0.50	0.51
query6	1.14	0.74	0.72
query7	0.02	0.03	0.02
query8	0.03	0.03	0.03
query9	0.58	0.50	0.50
query10	0.56	0.54	0.57
query11	0.14	0.09	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	3.08	2.94	3.00
query15	0.91	0.82	0.82
query16	0.38	0.39	0.39
query17	1.07	1.05	1.08
query18	0.23	0.21	0.21
query19	1.88	1.80	1.95
query20	0.01	0.01	0.02
query21	15.38	0.57	0.59
query22	2.63	2.49	2.10
query23	17.04	0.94	0.72
query24	2.99	0.52	1.54
query25	0.21	0.06	0.05
query26	0.42	0.13	0.15
query27	0.04	0.03	0.05
query28	10.72	1.10	1.06
query29	12.56	3.24	3.22
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.24	0.47	0.45
query33	2.98	3.08	3.01
query34	17.01	4.54	4.48
query35	4.51	4.51	4.46
query36	0.67	0.47	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.24 s
Total hot run time: 32.56 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@liaoxin01 @Thearas @doris-robot