Support getting response body for network responses #747
Labels
enhancement
New feature or request
module-network
Network module
needs-discussion
Issues to be discussed by the working group
Comments
42 tasks
3 tasks
whimboo
changed the title
3 tasks
whimboo
changed the title
|
The Browser Testing and Tools Working Group just discussed The full IRC log of that discussion<tidoust> Topic: Support getting response body for network responses<jgraham> github: https://github.com//issues/747 <tidoust> jgraham: Use case is that people want to be able to initially read and then read and write the body of HTTP responses. <tidoust> ... The difficulty is that these bodies may be rather large, so we can't send them fully over the wire. <tidoust> ... We talked before about getting something like streaming, with some method to pull more data until it's complete. <tidoust> ... The streaming approach seems necessary in at least some cases. <tidoust> ... The request interception is already in place, but this thing is still missing. My understanding is that this has become the remaining "blocker". <tidoust> ... Other thoughts on what the design should be here? Plans to work on this? <jdescottes> q+ <tidoust> ... We'll eventually end up doing it, but it anybody else is looking into it, that would be better. <sadym> q+ <tidoust> ack jdescottes <tidoust> jdescottes: Question for Maksym. Any API in Puppeteer that exposes response bodies as a stream? <tidoust> sadym: I had the impression that responseBuffer would return chunks, but it probably returns the whole thing at once. <tidoust> ack sadym <tidoust> sadym: I'm not sure if Alex is going to work on it soon or not, so cannot give you an estimate. <jgraham> Maybe https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-getResponseBodyForInterception allows getting the body as a string for intercepted requests <sadym> https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-getResponseBody <tidoust> jgraham: It does look like CDP has both full body and chunk streaming capabilities, through getResponseBody and getResponseBodyForInterception <tidoust> sadym: The only way it works is no chunk, but streams and base64 encoded streams. <tidoust> jgraham: That suggests some possibility that we could start by returning the whole body as a string, which would certainly help us ship something faster. <tidoust> ... It feels a good first step, but also scary if the body is a 2GB piece of data. <tidoust> sadym: Currently, our implementation only provides a full body, even though the API offers an apparent stream. It may lead to a false "safe" feeling for the user where they feel they get a chunk of the data whereas we sent the entire thing under the hoods. <tidoust> ... It may be better to stick to strings initially. <tidoust> ... and thus the whole body. <tidoust> jgraham: It seems there's some consensus to start with an API that gives you the full body. <jimevans> q+ <tidoust> ... A possible complication compared with fetch is that chunks may need to be split across messages. <tidoust> ack jimevans <tidoust> jimevans: Are we talking about a base64 encoded version of the body string? I worry about things such as people intercepting an image or any binary data. <tidoust> jgraham: We need to support both. If it's a thing with a text content-type, we should return a string. If not, I'm not sure about the details but we need to support both use cases (base64 encoding and binary), and there's a pattern on how to do that. <jgraham> RRSAgent: make minutes <RRSAgent> I have made the request to generate https://www.w3.org/2025/01/08-webdriver-minutes.html jgraham <jgraham> zakim, bye <Zakim> leaving. As of this point the attendees have been jimevans, jgraham, jugglinmike, lauromoura, sasha, Francois, benchatt, whimboo, jdescottes, simons <jgraham> RRSAgent, bye <RRSAgent> I see no action items |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue #541 asks for a mechanism to alter the response body when intercepting a request. This issue on the other hand is specifically about reading the body of network responses in responseCompleted events with or without the interception.
In Puppeteer, it is exposed via https://pptr.dev/api/puppeteer.httpresponse.buffer and https://pptr.dev/api/puppeteer.httpresponse.json
Probably adding support for fetching response bodies would require adding a new method and a way to deal with blobs in the protocol.
The text was updated successfully, but these errors were encountered: