Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

branch-3.0: [opt](nereids)prune unused column after push down common column from agg #46627 #47029

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from
Open

branch-3.0: [opt](nereids)prune unused column after push down common column from agg #46627 #47029

wants to merge 1 commit into from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #46627

@englefly
[opt](nereids)prune unused column after push down common column from …
7763f00 
…agg (#46627)

### What problem does this PR solve?

after extracting common expressions for agg, the underlying projection
may project redundant columns.
for example:
original plan 
Agg(groupkey=[A+B, A+B+1])
--> project(A, B)

after extracting, "A+B as C" is detected as a common expression, and the
plan becomes
Agg(groupKey=[C, C+1])
 -->project(A, B, A+B as C)

here A, B should not be projected, since they are not used any more. so
the optimal plan is
Agg(groupKey=[C, C+1])
 -->project(A+B as C)

Related PR: #40473
@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jan 15, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40205 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7763f0022d80a05cf443857deb6b5bd726f40e01, data reload: false

------ Round 1 ----------------------------------
q1	17598	6751	6643	6643
q2	2062	170	155	155
q3	10819	1080	1137	1080
q4	10413	771	793	771
q5	7742	2838	2803	2803
q6	219	134	134	134
q7	983	619	599	599
q8	9349	1945	2002	1945
q9	6594	6423	6397	6397
q10	7046	2279	2230	2230
q11	457	259	273	259
q12	398	212	209	209
q13	17788	3012	3038	3012
q14	236	212	208	208
q15	502	472	466	466
q16	679	606	602	602
q17	975	541	576	541
q18	7237	6716	6733	6716
q19	1391	1067	1022	1022
q20	476	202	196	196
q21	4021	3215	3301	3215
q22	1102	1002	1002	1002
Total cold run time: 108087 ms
Total hot run time: 40205 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6571	6573	6661	6573
q2	329	236	232	232
q3	2916	2836	2996	2836
q4	2118	1803	1831	1803
q5	5857	5840	5819	5819
q6	219	129	128	128
q7	2261	1818	1874	1818
q8	3344	3554	3516	3516
q9	8905	8855	8872	8855
q10	3503	3476	3440	3440
q11	593	479	488	479
q12	818	614	613	613
q13	9241	3135	3192	3135
q14	305	291	286	286
q15	528	470	482	470
q16	718	681	661	661
q17	1843	1593	1609	1593
q18	8308	7840	7716	7716
q19	1681	1660	1467	1467
q20	2036	1853	1856	1853
q21	5492	5269	5394	5269
q22	1147	1113	1051	1051
Total cold run time: 68733 ms
Total hot run time: 59613 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196511 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7763f0022d80a05cf443857deb6b5bd726f40e01, data reload: false

query1	1284	897	888	888
query2	6243	2135	2118	2118
query3	10965	4378	4298	4298
query4	61398	28855	23309	23309
query5	5082	463	460	460
query6	415	181	186	181
query7	5438	319	303	303
query8	307	229	222	222
query9	8460	2598	2582	2582
query10	445	269	258	258
query11	17394	14976	15879	14976
query12	157	105	105	105
query13	1411	450	458	450
query14	10309	7480	6900	6900
query15	214	187	176	176
query16	7156	498	507	498
query17	1147	603	632	603
query18	1780	329	319	319
query19	212	162	156	156
query20	119	115	115	115
query21	212	105	103	103
query22	4741	4409	4946	4409
query23	34558	34267	33748	33748
query24	6128	2910	3004	2910
query25	566	433	441	433
query26	651	173	167	167
query27	1863	358	353	353
query28	4312	2462	2470	2462
query29	767	450	436	436
query30	237	162	166	162
query31	984	818	782	782
query32	63	60	55	55
query33	435	286	281	281
query34	916	513	523	513
query35	865	723	720	720
query36	1079	955	990	955
query37	127	76	73	73
query38	4133	4080	4015	4015
query39	1514	1486	1453	1453
query40	201	96	99	96
query41	50	49	49	49
query42	120	102	101	101
query43	544	500	507	500
query44	1158	838	824	824
query45	186	164	173	164
query46	1196	743	732	732
query47	2019	1963	1934	1934
query48	476	396	373	373
query49	732	378	399	378
query50	813	423	424	423
query51	7354	7172	7183	7172
query52	98	87	92	87
query53	259	179	185	179
query54	569	477	463	463
query55	79	80	77	77
query56	261	241	257	241
query57	1257	1165	1143	1143
query58	219	214	219	214
query59	3457	3098	2940	2940
query60	284	255	252	252
query61	108	107	114	107
query62	744	601	632	601
query63	218	190	189	189
query64	1362	664	672	664
query65	3245	3212	3194	3194
query66	728	303	294	294
query67	15960	15867	15707	15707
query68	4124	591	571	571
query69	437	269	265	265
query70	1212	1111	1147	1111
query71	367	277	259	259
query72	6351	4037	4003	4003
query73	756	351	371	351
query74	10109	8921	8927	8921
query75	3339	2637	2650	2637
query76	1978	1227	1049	1049
query77	506	277	265	265
query78	10684	9660	9585	9585
query79	2097	590	617	590
query80	1315	424	437	424
query81	521	235	242	235
query82	1254	111	116	111
query83	269	147	139	139
query84	283	74	74	74
query85	970	295	293	293
query86	404	303	277	277
query87	4386	4410	4335	4335
query88	3821	2393	2371	2371
query89	412	301	302	301
query90	1934	186	187	186
query91	181	146	151	146
query92	64	50	48	48
query93	2691	539	536	536
query94	766	304	296	296
query95	355	263	251	251
query96	627	290	288	288
query97	3346	3189	3246	3189
query98	221	205	199	199
query99	1476	1172	1160	1160
Total cold run time: 315040 ms
Total hot run time: 196511 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7763f0022d80a05cf443857deb6b5bd726f40e01, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.53	0.52	0.51
query6	1.14	0.74	0.72
query7	0.02	0.02	0.02
query8	0.03	0.04	0.04
query9	0.55	0.49	0.52
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	2.91	3.02	2.95
query15	0.93	0.83	0.83
query16	0.37	0.38	0.38
query17	1.06	1.02	1.05
query18	0.23	0.22	0.22
query19	1.88	1.87	2.02
query20	0.01	0.01	0.01
query21	15.35	0.59	0.57
query22	2.24	2.82	2.18
query23	16.84	0.87	0.94
query24	3.04	1.90	1.48
query25	0.29	0.15	0.06
query26	0.46	0.14	0.15
query27	0.05	0.04	0.05
query28	9.22	0.55	0.45
query29	12.58	3.29	3.30
query30	0.25	0.06	0.06
query31	2.86	0.40	0.38
query32	3.23	0.46	0.46
query33	3.00	2.99	3.01
query34	17.05	4.56	4.52
query35	4.55	4.51	4.56
query36	0.67	0.48	0.51
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.23 s
Total hot run time: 33.44 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Thearas @doris-robot @dataroaring @englefly