-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JCR-4369: Avoid S3 Incomplete Read Warning by elegant aborting #61
Open
woonsan
wants to merge
7,994
commits into
apache:trunk
Choose a base branch
from
woonsan:feature/JCR-4369
base: trunk
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1777759 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1777949 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1777951 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1778270 13f79535-47bb-0310-9956-ffa450edef68
…de() git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1779166 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1779460 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1779614 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1779632 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1780208 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1780220 13f79535-47bb-0310-9956-ffa450edef68
…and minin type based inheritance. Currently it appears that only primaryType based inheritance is working git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1780335 13f79535-47bb-0310-9956-ffa450edef68
…and minin type based inheritance. Currently it appears that only primaryType based inheritance is working git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1780336 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1780384 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1780406 13f79535-47bb-0310-9956-ffa450edef68
…ts initializing git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1782145 13f79535-47bb-0310-9956-ffa450edef68
…ts initializing git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1782173 13f79535-47bb-0310-9956-ffa450edef68
- Using SHA-256 instead of SHA-1 to create blob record ids git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1785225 13f79535-47bb-0310-9956-ffa450edef68
Candidate Release Notes git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1785264 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1785460 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1785462 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1785579 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1785580 13f79535-47bb-0310-9956-ffa450edef68
create valid names / do sanity check in test case git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1786325 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1786330 13f79535-47bb-0310-9956-ffa450edef68
…pl.fixLegacy() git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1787043 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1787381 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1792066 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1792100 13f79535-47bb-0310-9956-ffa450edef68
maven-scr-plugin git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1792105 13f79535-47bb-0310-9956-ffa450edef68
maven-assembly-plugin git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1792113 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836579 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836620 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836624 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836635 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836637 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836722 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836729 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836731 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836733 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836773 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836776 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836778 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836794 13f79535-47bb-0310-9956-ffa450edef68
…pendency git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1836800 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1837360 13f79535-47bb-0310-9956-ffa450edef68
… API Patch provided by Alexander Klimetschek, with review and contribution from Julian Reschke git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1838616 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839647 13f79535-47bb-0310-9956-ffa450edef68
…rsistence Manager (patch by Woonsan Ko from #60) git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839660 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839910 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839924 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839931 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839937 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839945 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839964 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1839966 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1840007 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/jackrabbit/trunk@1840027 13f79535-47bb-0310-9956-ffa450edef68
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AWS S3 SDK recommends to abort the
S3ObjectInputStream
if not intended to consume the data because the http connection pool behavior (by consuming data on close to reuse the connection) of HttpClient under the hood could cause a big performance issue when reading big amount of data. By aborting, it's better to simply abort the underlyingHttpRequestBase
and kick out the connection from the pool from AWS S3 SDK perspective.In multi-threaded working environment (due to multiple requests and/or
proactiveCaching
mode ofCachingDataStore
), the reading and storing actions ino.a.j.c.data.CachingDataStore.getStream(DataIdentifier)
results in falling in the else block ofo.a.j.core.data.LocalCache.store(String, InputStream)
while the file by the name could already exist when executing the else block. In that case, theS3ObjectInputStream
is never read and aborted. As a result,com.amazonaws.services.s3.internal.S3AbortableInputStream#close()
ends up complaining about non-aborted/non-read-fully input stream.Therefore, my fix includes the following:
LocalCache
checks if the backend resource input stream is abortable. If abortable, it tries to abort the backend resource stream. For this purpose,BackendResourceAbortable
interface in jackrabbit-data is introduced.S3Backend
wraps theS3ObjectInputStream
to implementBackendResourceAbortable
by leveraging commons-io'sProxyInputStream
.