Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](metrics) Correct statistical method for file cache stats #47020

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

TangSiyang2001
Copy link
Collaborator

What problem does this PR solve?

Problem Summary:

Current file cache statistics in not calculate in right way. It calculate reading bytes num by accumulate reading bytes in current io context repeadly rather than accumulate by increment. Correct this behavior in this PR.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@TangSiyang2001
Copy link
Collaborator Author

run buildall

@TangSiyang2001 TangSiyang2001 force-pushed the fix-file-cache-read-stats branch from 2ed30df to 2d905e2 Compare January 15, 2025 06:58
@TangSiyang2001
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33064 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d905e2723119174c40288313dbb0646e6793641, data reload: false

------ Round 1 ----------------------------------
q1	17585	6152	6113	6113
q2	2047	296	168	168
q3	10417	1306	731	731
q4	10222	970	513	513
q5	7516	2396	2194	2194
q6	206	178	149	149
q7	906	758	610	610
q8	9268	1354	1147	1147
q9	5285	4902	4948	4902
q10	6833	2376	1939	1939
q11	480	277	259	259
q12	350	355	219	219
q13	17783	3747	3074	3074
q14	246	236	207	207
q15	566	530	519	519
q16	634	624	575	575
q17	566	855	321	321
q18	6979	6515	6402	6402
q19	1213	960	535	535
q20	317	323	200	200
q21	2995	2208	1984	1984
q22	362	335	303	303
Total cold run time: 102776 ms
Total hot run time: 33064 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6205	6229	6230	6229
q2	237	333	242	242
q3	2295	2618	2350	2350
q4	1385	1816	1390	1390
q5	4369	4794	4824	4794
q6	183	176	143	143
q7	2101	1985	1860	1860
q8	2608	2827	2752	2752
q9	7176	7232	7210	7210
q10	3025	3359	2824	2824
q11	588	518	501	501
q12	677	779	665	665
q13	3485	3912	3265	3265
q14	286	300	290	290
q15	589	533	513	513
q16	645	686	642	642
q17	1218	1776	1278	1278
q18	7707	7549	7272	7272
q19	785	1121	1076	1076
q20	2034	2039	1917	1917
q21	5757	5345	4879	4879
q22	598	597	537	537
Total cold run time: 53953 ms
Total hot run time: 52629 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190252 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d905e2723119174c40288313dbb0646e6793641, data reload: false

query1	950	390	383	383
query2	6530	2113	2141	2113
query3	6810	217	214	214
query4	34110	23419	23283	23283
query5	4307	617	489	489
query6	293	208	203	203
query7	4610	494	300	300
query8	275	237	225	225
query9	9771	2701	2671	2671
query10	473	305	246	246
query11	18201	15178	15429	15178
query12	149	110	104	104
query13	1636	518	379	379
query14	9962	7537	7496	7496
query15	246	198	196	196
query16	7999	653	502	502
query17	1629	750	573	573
query18	2116	405	307	307
query19	222	182	160	160
query20	117	153	108	108
query21	204	122	105	105
query22	4291	4578	4540	4540
query23	33932	33535	33163	33163
query24	6520	2286	2330	2286
query25	463	443	371	371
query26	1069	226	170	170
query27	2020	468	325	325
query28	5408	2430	2397	2397
query29	561	504	423	423
query30	232	197	162	162
query31	977	886	803	803
query32	103	66	59	59
query33	508	347	301	301
query34	740	848	505	505
query35	806	836	750	750
query36	979	1064	940	940
query37	119	103	77	77
query38	4173	4176	4069	4069
query39	1442	1401	1371	1371
query40	208	119	99	99
query41	54	51	50	50
query42	119	102	103	102
query43	507	538	485	485
query44	1368	834	830	830
query45	180	175	173	173
query46	887	1064	651	651
query47	1805	1823	1747	1747
query48	380	404	330	330
query49	801	490	392	392
query50	659	658	390	390
query51	7016	6869	6916	6869
query52	106	102	96	96
query53	228	248	186	186
query54	500	482	417	417
query55	83	81	81	81
query56	286	296	283	283
query57	1182	1180	1100	1100
query58	258	241	247	241
query59	3043	3114	3003	3003
query60	294	291	274	274
query61	152	140	140	140
query62	898	776	732	732
query63	232	196	189	189
query64	4307	1019	694	694
query65	3229	3191	3190	3190
query66	1045	444	370	370
query67	15748	15731	15475	15475
query68	5633	830	535	535
query69	489	300	251	251
query70	1195	1163	1152	1152
query71	377	287	249	249
query72	5694	3863	3937	3863
query73	661	761	372	372
query74	10357	9106	8863	8863
query75	3197	3145	2651	2651
query76	3203	1184	791	791
query77	551	379	276	276
query78	9990	10305	9408	9408
query79	2710	840	613	613
query80	624	522	429	429
query81	520	329	238	238
query82	355	156	122	122
query83	175	179	169	169
query84	245	100	72	72
query85	765	338	298	298
query86	393	314	311	311
query87	4689	4667	4494	4494
query88	4861	2197	2178	2178
query89	389	343	301	301
query90	1893	194	193	193
query91	135	137	110	110
query92	72	62	62	62
query93	1566	854	535	535
query94	670	411	299	299
query95	343	284	260	260
query96	493	614	281	281
query97	2799	2884	2759	2759
query98	225	201	201	201
query99	1406	1524	1380	1380
Total cold run time: 287203 ms
Total hot run time: 190252 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d905e2723119174c40288313dbb0646e6793641, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.08	0.07
query4	1.61	0.10	0.11
query5	0.43	0.42	0.44
query6	1.20	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.49	0.50
query10	0.55	0.57	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.59	0.59
query14	2.81	2.74	2.73
query15	0.90	0.83	0.83
query16	0.38	0.41	0.39
query17	1.08	1.00	1.07
query18	0.24	0.21	0.20
query19	1.89	1.88	1.96
query20	0.01	0.02	0.01
query21	15.37	0.93	0.59
query22	0.75	0.78	0.62
query23	15.37	1.38	0.52
query24	3.26	1.47	1.65
query25	0.24	0.11	0.06
query26	0.18	0.14	0.14
query27	0.06	0.05	0.06
query28	14.54	1.50	1.04
query29	12.57	4.02	3.26
query30	0.25	0.09	0.06
query31	2.84	0.60	0.39
query32	3.25	0.55	0.46
query33	3.05	2.98	3.02
query34	16.71	5.15	4.52
query35	4.53	4.49	4.49
query36	0.65	0.50	0.47
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.09 s
Total hot run time: 31.67 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.01% (10687/26059)
Line Coverage: 31.52% (90287/286467)
Region Coverage: 30.68% (46259/150797)
Branch Coverage: 26.84% (23445/87360)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d905e2723119174c40288313dbb0646e6793641_2d905e2723119174c40288313dbb0646e6793641/report/index.html

@TangSiyang2001
Copy link
Collaborator Author

run p0

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 15, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants