Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: retry timeout errors #34284

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/api/class-requestoptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Defaults to `20`. Pass `0` to not follow redirects.
* since: v1.46
- `maxRetries` <[int]>

Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the timeout errors that are retried? Does it now mean that overall timeout is maxRetries x timeout? I'm not sure I like the latter. If we want to support this, we could just add ETIMEDOUT (socket timeouts) to the retryable error codes and keep old semantics of the timeout parameter, i.e. if ETIMEDOUT occurs and the deadline is not reached yet (and retries count < maxRetries), we will retry, otherwise we bail out. I don't think we want to retry errors where the server received the request but was got stuck processing it and was not able to finish its response on time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that makes sense. fiddling with the deadline parameter already felt wrong to me. i'll rewrite it to only ETIMEDOUT.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After looking into this more, I think that won't work. ETIMEDOUT is the error that occurs when a socket idles for a specified amount of time. By default, there's no timeout. So unless we add API for configuring socket timeout, we won't be able to retry ETIMEDOUT.

Most other forms of timeout i've tested show up as ECONNRESET, so they're already being retried today.

You're saying "we don't want to retry when the server got stuck processing", but given this I'm pretty sure that's what the user report is about. Let's discuss this more tonight.


## method: RequestOptions.setMethod
* since: v1.18
Expand Down
2 changes: 1 addition & 1 deletion docs/src/api/class-route.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,7 @@ Defaults to `20`. Pass `0` to not follow redirects.
* since: v1.46
- `maxRetries` <[int]>

Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.

### option: Route.fetch.timeout
* since: v1.33
Expand Down
2 changes: 1 addition & 1 deletion docs/src/api/params.md
Original file line number Diff line number Diff line change
Expand Up @@ -494,7 +494,7 @@ Defaults to `20`. Pass `0` to not follow redirects.
* langs: js, python, csharp
- `maxRetries` <[int]>

Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.

## evaluate-expression
- `expression` <[string]>
Expand Down
37 changes: 25 additions & 12 deletions packages/playwright-core/src/server/fetch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,17 @@ export type APIRequestFinishedEvent = {

type SendRequestOptions = https.RequestOptions & {
maxRedirects: number,
deadline: number,
deadline?: number, // applies to the whole retry sequence.
headers: HeadersObject,
__testHookLookup?: (hostname: string) => LookupAddress[]
};

class RequestTimedOutError extends Error {
constructor(timeout: number) {
super(`Request timed out after ${timeout}ms`);
}
}

export abstract class APIRequestContext extends SdkObject {
static Events = {
Dispose: 'dispose',
Expand Down Expand Up @@ -180,15 +186,13 @@ export abstract class APIRequestContext extends SdkObject {


const timeout = defaults.timeoutSettings.timeout(params);
const deadline = timeout && (monotonicTime() + timeout);

const options: SendRequestOptions = {
method,
headers,
agent,
maxRedirects: params.maxRedirects === 0 ? -1 : params.maxRedirects === undefined ? 20 : params.maxRedirects,
timeout,
deadline,
...getMatchingTLSOptionsForOrigin(this._defaultOptions().clientCertificates, requestUrl.origin),
__testHookLookup: (params as any).__testHookLookup,
};
Expand Down Expand Up @@ -255,24 +259,33 @@ export abstract class APIRequestContext extends SdkObject {
}
}

private async _sendRequestWithRetries(progress: Progress, url: URL, options: SendRequestOptions, postData?: Buffer, maxRetries?: number): Promise<Omit<channels.APIResponse, 'fetchUid'> & { body: Buffer }>{
maxRetries ??= 0;
private async _sendRequestWithRetries(progress: Progress, url: URL, options: SendRequestOptions, postData?: Buffer, maxRetries = 0): Promise<Omit<channels.APIResponse, 'fetchUid'> & { body: Buffer }>{
let backoff = 250;
for (let i = 0; i <= maxRetries; i++) {
try {
return await this._sendRequest(progress, url, options, postData);
const requestOptions = { ...options };
if (options.timeout)
requestOptions.deadline ??= monotonicTime() + options.timeout;
return await this._sendRequest(progress, url, requestOptions, postData);
} catch (e) {
e = rewriteOpenSSLErrorIfNeeded(e);
if (maxRetries === 0)
throw e;
if (i === maxRetries || (options.deadline && monotonicTime() + backoff > options.deadline))
throw new Error(`Failed after ${i + 1} attempt(s): ${e}`);
// Retry on connection reset only.
if (e.code !== 'ECONNRESET')

async function retry(reason: string) {
progress.log(` ${reason}, will retry after ${backoff}ms.`);
await new Promise(f => setTimeout(f, backoff));
backoff *= 2;
}

if (e instanceof RequestTimedOutError)
await retry('Request timed out');
else if (e.code === 'ECONNRESET')
await retry('Received ECONNRESET');
else
throw e;
progress.log(` Received ECONNRESET, will retry after ${backoff}ms.`);
await new Promise(f => setTimeout(f, backoff));
backoff *= 2;
}
}
throw new Error('Unreachable');
Expand Down Expand Up @@ -544,7 +557,7 @@ export abstract class APIRequestContext extends SdkObject {

if (options.deadline) {
const rejectOnTimeout = () => {
reject(new Error(`Request timed out after ${options.timeout}ms`));
reject(new RequestTimedOutError(options.timeout!));
request.destroy();
};
const remaining = options.deadline - monotonicTime();
Expand Down
40 changes: 24 additions & 16 deletions packages/playwright-core/types/types.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17670,8 +17670,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -17801,8 +17802,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -17919,8 +17921,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -18005,8 +18008,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -18091,8 +18095,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -18219,8 +18224,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -18305,8 +18311,9 @@ export interface APIRequestContext {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down Expand Up @@ -20867,8 +20874,9 @@ export interface Route {
maxRedirects?: number;

/**
* Maximum number of times network errors should be retried. Currently only `ECONNRESET` error is retried. Does not
* retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no retries.
* Maximum number of times network errors should be retried. Only `ECONNRESET` and timeout errors are retried. Does
* not retry based on HTTP response codes. An error will be thrown if the limit is exceeded. Defaults to `0` - no
* retries.
*/
maxRetries?: number;

Expand Down
18 changes: 18 additions & 0 deletions tests/library/global-fetch.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -536,3 +536,21 @@ it('should retry ECONNRESET', {
expect(requestCount).toBe(4);
await request.dispose();
});

it('should retry ETIMEDOUT', {
annotation: { type: 'issue', description: 'https://github.com/microsoft/playwright/issues/34207' }
}, async ({ playwright, server }) => {
const request = await playwright.request.newContext();
let requestCount = 0;
server.setRoute('/test', (req, res) => {
if (requestCount++ < 3)
return;
res.writeHead(200, { 'content-type': 'text/plain' });
res.end('Hello!');
});
const response = await request.fetch(server.PREFIX + '/test', { maxRetries: 3, timeout: 100 });
expect(response.status()).toBe(200);
expect(await response.text()).toBe('Hello!');
expect(requestCount).toBe(4);
await request.dispose();
});
Loading