chore(bidi): add support for fetching response bodies #37244

hbenl · 2025-08-29T14:40:09Z

Fixes #37218 and (at least) the following tests in Firefox:

tests/library/browsercontext-base-url.spec.ts:
- "should be able to match a URL relative to its given URL with urlMatcher"
tests/library/browsercontext-credentials.spec.ts:
- "should return resource body"
tests/library/browsercontext-har.spec.ts:
- "should update extracted har.zip for page"
- "should update har.zip for context"
- "should update har.zip for page"
- "should update har.zip for page with different options"
tests/library/browsertype-connect.spec.ts:
- "launchServer > should fulfill with global fetch result"
- "run-server > should fulfill with global fetch result"
tests/library/har.spec.ts:
- "should attach content"
- "should contain http2 for http2 requests"
- "should include content smoke"
- "should use attach mode for zip extension"
tests/page/frame-goto.spec.ts:
- "should return matching responses"
tests/page/page-event-request.spec.ts:
- "should return response body when Cross-Origin-Opener-Policy is set"
tests/page/page-history.spec.ts:
- "page.reload should not resolve with same-document navigation"
tests/page/page-network-response.spec.ts:
- "should return text"
- "should return uncompressed text"
- "should wait until response completes"
- "should return json"
- "should return body"
- "should return body with compression"
- "should return body for prefetch script"
tests/page/page-request-fallback.spec.ts:
- "should not chain fulfill"
- "should chain once"
tests/page/page-request-fulfill.spec.ts:
- "should fulfill with global fetch result"
- "should fulfill with fetch result"
- "should fulfill with fetch result and overrides"
- "should return body for fulfilled responses"
tests/page/page-request-intercept.spec.ts:
- "should fulfill with any response"
- "should intercept with url override"
- "should support fulfill after intercept"
tests/page/workers.spec.ts:
- "should report worker script as network request"
- "should report worker script as network request after redirect"

yury-s · 2025-09-08T17:47:27Z

packages/playwright-core/src/server/bidi/bidiPage.ts

  }

+  async updateNetworkDataCollection() {
+    await this._networkManager.setDataCollection(true);


nit: just inline this call as there is only one call site of the method.

yury-s · 2025-09-08T17:51:54Z

packages/playwright-core/src/server/bidi/bidiNetworkManager.ts

+    if (enable) {
+      if (!this._dataCollectorId) {
+        const dataTypes: [bidi.Network.DataType] = [bidi.Network.DataType.Response];
+        const maxEncodedDataSize = maxSize ?? 10_000_000;


Let's make it 200Mb to match Chromium constant.

I've set it to 20_000_000 as per this comment.

Let's add a comment pointing at https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/inspector/inspector_network_agent.cc;l=134;drc=4128411589187a396829a827f59a655bed876aa7 as the source of the number.

yury-s · 2025-09-08T17:53:20Z

packages/playwright-core/src/server/bidi/bidiNetworkManager.ts

+        this._dataCollectorId = collector;
+      }
+    } else {
+      if (this._dataCollectorId) {


This branch is never reached, let's drop it.

yury-s · 2025-09-08T17:56:48Z

packages/playwright-core/src/server/bidi/bidiNetworkManager.ts

    await this._updateProtocolRequestInterception();
  }

+  async setDataCollection(enable: boolean, maxSize?: number) {


Let's rename the method into initialize() and remove all the if checks, we want the resource collection to be be enabled and done so during the setup phase.

Also if we don't have to pass collector to network.getData, there is no need to store in the network manager. Let's keep only the code/data that is actually used.

yury-s · 2025-09-08T17:58:28Z

packages/playwright-core/src/server/bidi/bidiNetworkManager.ts

      return;
    const getResponseBody = async () => {
-      throw new Error(`Response body is not available for requests in Bidi`);
+      const { bytes } = await this._session.send('network.getData', { request: params.request.request, dataType: bidi.Network.DataType.Response });


Do we need to also pass collector parameter?

yury-s

Heads-up, #37321 is adding the same functionality.

yury-s · 2025-09-09T19:29:53Z

packages/playwright-core/src/server/bidi/bidiNetworkManager.ts

+    const dataTypes: [bidi.Network.DataType] = [bidi.Network.DataType.Response];
+    const maxEncodedDataSize = 20_000_000;
+    const contexts: [string] = [this._session.sessionId];
+    await this._session.send('network.addDataCollector', { dataTypes, maxEncodedDataSize, contexts });


Is there a reason we do it per page, rather than initialize once per browser session as in https://github.com/microsoft/playwright/pull/37321/files#diff-8753d59ba95abdc3367549f4caf85cbe4fa7b702d4f94427301bc9eadf4e79e1 ?

No particular reason, I've updated the PR to do it once per session.

github-actions · 2025-09-10T16:28:45Z

Test results for "tests 1"

5 flaky

⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors `@firefox-ubuntu-22.04-node18`
⚠️ [firefox-library] › library/inspector/cli-codegen-pick-locator.spec.ts:35 › should update locator highlight `@firefox-ubuntu-22.04-node18`
⚠️ [chromium-library] › library/chromium/oopif.spec.ts:374 › should allow to re-connect to OOPIFs with CDP when iframes were there already `@ubuntu-22.04-chromium-tip-of-tree`
⚠️ [playwright-test] › ui-mode-test-output.spec.ts:80 › should show console messages for test `@ubuntu-latest-node22-1`
⚠️ [playwright-test] › ui-mode-trace.spec.ts:637 › should indicate current test status `@ubuntu-latest-node24-1`

45357 passed, 816 skipped

Merge workflow run.

This comment has been minimized.

Sign in to view

yury-s approved these changes Sep 8, 2025

View reviewed changes

yury-s mentioned this pull request Sep 8, 2025

chore(bidi): implement response bodies #37321

Closed

yury-s reviewed Sep 8, 2025

View reviewed changes

hbenl force-pushed the response-bodies branch from aab8eb0 to d1a031a Compare September 9, 2025 08:25

This comment has been minimized.

Sign in to view

yury-s approved these changes Sep 9, 2025

View reviewed changes

chore(bidi): add support for fetching response bodies

a6992d1

hbenl force-pushed the response-bodies branch from d1a031a to a6992d1 Compare September 10, 2025 10:27

yury-s approved these changes Sep 10, 2025

View reviewed changes

yury-s merged commit 4dd1949 into microsoft:main Sep 10, 2025
34 of 37 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore(bidi): add support for fetching response bodies #37244

chore(bidi): add support for fetching response bodies #37244

Uh oh!

hbenl commented Aug 29, 2025

Uh oh!

This comment has been minimized.

yury-s Sep 8, 2025

Uh oh!

yury-s Sep 8, 2025

Uh oh!

hbenl Sep 9, 2025

Uh oh!

yury-s Sep 9, 2025

Uh oh!

yury-s Sep 8, 2025

Uh oh!

yury-s Sep 8, 2025

Uh oh!

yury-s Sep 8, 2025

Uh oh!

yury-s left a comment

Uh oh!

This comment has been minimized.

yury-s Sep 9, 2025

Uh oh!

hbenl Sep 10, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 10, 2025

Uh oh!

Uh oh!

chore(bidi): add support for fetching response bodies #37244

chore(bidi): add support for fetching response bodies #37244

Uh oh!

Conversation

hbenl commented Aug 29, 2025

Uh oh!

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yury-s left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Sep 10, 2025

Test results for "tests 1"

Uh oh!

Uh oh!