Proof of concept: test configuration management

[carli2]

I made this fork to demonstrate a proof of concept for test configuration management.

The problem is as follows:

Deployment URLs of test instances can change as well as credentials like usernames and passwords as well as other information that will depend on the test run setting.

This fork tries to integrate support for test configuration management directly into the code generator. I implemented the following features:

• activate configuration management by passing npx playwright --include-conf=test/conf.json http://url-to-test
• the following line is added to the test runner: const conf = JSON.parse(require('fs').readFileSync('test/conf.json'));
• instead of hardcoding page.goto() URLs, the url-to-test is stored in conf.baseurl, thus we have await page.goto(conf.baseurl + '/'); as the first line
• instead of hardcoding credentials, the code generator will write now: await page.locator('input[name="username"]').fill(conf.username); when the field name and field content matches the one in the config

I don't expect you to merge that pull request. But I want to give the code to you as a hint for further improvements.

[carli2]

@microsoft-github-policy-service agree [company="Launix, Inh. Carl-Philip Hänsch"]

[carli2]

@microsoft-github-policy-service agree company="Launix, Inh. Carl-Philip Hänsch"

[carli2]

to the maintainers: please review this PR anyway, it is just a proof-of-concept.

[yury-s]

Please start with filing a feature request that makes a case for the new functinality.

[carli2]

Please start with filing a feature request that makes a case for the new functinality.

#37271

[github-actions]

Test results for "tests 1"

48 failed
❌ [chromium-library] › library/debug-controller.spec.ts:250 › should record @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:43 › should print the correct context options when using a device @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @chromium-ubuntu-22.04-node18
❌ [chromium-library] › library/debug-controller.spec.ts:250 › should record @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:43 › should print the correct context options when using a device @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @chromium-ubuntu-22.04-node20
❌ [chromium-library] › library/debug-controller.spec.ts:250 › should record @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:43 › should print the correct context options when using a device @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @chromium-ubuntu-22.04-node22
❌ [chromium-library] › library/debug-controller.spec.ts:250 › should record @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:43 › should print the correct context options when using a device @chromium-ubuntu-22.04-node24
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @chromium-ubuntu-22.04-node24
❌ [firefox-library] › library/debug-controller.spec.ts:250 › should record @firefox-ubuntu-22.04-node18
❌ [firefox-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @firefox-ubuntu-22.04-node18
❌ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @firefox-ubuntu-22.04-node18
❌ [firefox-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @firefox-ubuntu-22.04-node18
❌ [firefox-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @firefox-ubuntu-22.04-node18
❌ [firefox-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @firefox-ubuntu-22.04-node18
❌ [chromium-library] › library/debug-controller.spec.ts:250 › should record @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:43 › should print the correct context options when using a device @ubuntu-22.04-chromium-tip-of-tree
❌ [chromium-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @ubuntu-22.04-chromium-tip-of-tree
❌ [webkit-library] › library/debug-controller.spec.ts:250 › should record @webkit-ubuntu-22.04-node18
❌ [webkit-library] › library/debug-controller.spec.ts:285 › should record custom data-testid @webkit-ubuntu-22.04-node18
❌ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:1079 › cli codegen › should not throw csp directive violation errors @webkit-ubuntu-22.04-node18
❌ [webkit-library] › library/inspector/cli-codegen-test.spec.ts:20 › should print the correct imports and context options @webkit-ubuntu-22.04-node18
❌ [webkit-library] › library/inspector/cli-codegen-test.spec.ts:30 › should print the correct context options for custom settings @webkit-ubuntu-22.04-node18
❌ [webkit-library] › library/inspector/cli-codegen-test.spec.ts:57 › should print the correct context options when using a device and additional options @webkit-ubuntu-22.04-node18
❌ [webkit-library] › library/inspector/cli-codegen-test.spec.ts:72 › should print load storageState @webkit-ubuntu-22.04-node18

2 flaky

⚠️ [firefox-page] › page/page-wait-for-function.spec.ts:104 › should work with strict CSP policy `@firefox-ubuntu-22.04-node18`
⚠️ [playwright-test] › ui-mode-test-output.spec.ts:117 › should collapse repeated console messages for test `@ubuntu-latest-node22-1`

46728 passed, 821 skipped

---

Merge workflow run.

[yury-s]

Thanks for the proof of concept! We discussed possible improvements in this area during the team meeting today, I captured some of our thoughts in the linked issue. Since we don’t plan to introduce a .json config for the codegen, I’ll go ahead and close this PR.