feat: supports simultaneous upload and download rate limiting #634
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About this change - What it does
The functionality, which originally only limited uploads, has been changed to a global rate limit.
Resolves:
The purpose of this change is twofold:
When your remote storage service is located in a public cloud, such as AWS S3, and your server is on-premises, pulling data from the remote can easily saturate the local bandwidth, affecting the normal operation of other services.
Generally, AWS EC2's free disk throughput limit is 128MB. When uploading, downloading, and reading from local cache, without a global rate limiter to prevent competition between these operations, disk I/O can become uncontrollable, leading to issues with the producer's write operations.