[Feature] "Headless" CLI Watch Mode

[RichiCoder1]

Playwright recently introduced a watch mode via a new --ui flag. While awesome, it doesn't quite meet some other use cases for a "Watch Mode". Namely continuous background validation of tests where a UI might be intrusive, but you still want red/green when test srcs are updated. At that point you might jump in the UI to debug, but by default you're mostly just interested in continuous background validation.

Playwright is also increasingly being used for unit tests or integration tests, where a full browser step through might not be relevant.

[pavelfeldman]

Try setting environment variable PWTEST_WATCH=1 and running your tests. It'll engage a different version of the watch mode that should look more like the traditional jest one. Tell us what you think about it and whether it worked for you. We'll collect upvotes here to see if we want to make it official --watch.

[lucasassisrosa]

While we all waiting on this there's also a way to run a watch mode if you're on CI, using the git staged files. You can pull the files changed in git and feed to Playwright CLI to run only the specs you want. Here's an example with Github Actions:

steps:
  - name: Checkout
    uses: actions/checkout@v3

  - name: Setup NodeJS
    uses: actions/setup-node@v3
    with:
      node-version: 16

  - name: Install dependencies
    run: npm install

  - name: Install latest playwright browsers
    run: npx playwright install --with-deps

  - name: Get changed specs in the src folder
    id: changed-files-folder
    uses: tj-actions/changed-files@v35
    with:
      files: folder/**

  - name: Test 'e2e' if folder files changed
    if: steps.changed-files-folder.outputs.any_changed == 'true'
    run: npx playwright test ${{ steps.changed-files-folder.outputs.all_changed_files }}

at the end, the above would issue a command like

npx playwright test example.spec.ts second.spec.ts

[jfgreffier]

I really love PWTEST_WATCH=1 with component testing, it makes the experience more enjoyable than UI Mode, especially if you attempt to do TDD. Also the possibility to see the actual browser with the real component (instead of Trace) is a plus compared to the UI Mode.

[naveedausaf]

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

[pavelfeldman]

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

That's what it does.

[nausaf]

UPDATE: Now using onchange with -k option instead of chokidar. If there's a file change while tests are already running in response to a previous change, this would kill running tests and restart. chokidar on the other hand would launch tests another time in parallel.

This is the new test-playwright:watch script:

"test-playwright:testcommand": "(playwright test quiz.spec.ts || true) && echo Test run report available at http://localhost:6007",

"test-playwright:watch": "concurrently --names \"REPORT,TEST\" --hide REPORT --prefix-colors \"blue,magenta\" \"http-server playwright-report -s --port 6007\" \"onchange -i -k tests/**/*.spec.ts pages/* components/**/* -- npm run test-playwright:testcommand\"",

Hi @pavelfeldman,

I am pretty certain that watch mode of playwright test only monitors test files. I couldn't trigger it by making changes to any of my code files (React components) and saving the files. I also couldn't find any command line option to make it watch non-test code.

Another thing it doesn't do is create or serve HTML report that can be really useful when there are failures.

I am currently using a workaround using chokidar, playwright test, http-server etc. (given below). It would be great if the watch mode built into playwright test could incorporate these two features.

My current watch mode setup is as follows: I have a script named test-playwright:watch in package.json to do both (i.e. watch for changes to both test and code-under-test files, and to serve HTML report). It uses another two scripts and all three are shown below:

"test-playwright-watch-testcommand": "(playwright test quiz.spec.ts || true) && echo Test run report available at http://localhost:6007",

"test-playwright-watch-watchcommand": "chokidar tests/**/*.spec.ts pages/* components/**/* -c \"npm run test-playwright-watch-testcommand\"",

"test-playwright:watch": "concurrently -k -s first -n \"REPORT,TEST\" -c \"magenta,blue\" \"http-server playwright-report -s --port 6007\"  \"run-s --silent -c test-playwright-watch-testcommand test-playwright-watch-watchcommand\"",

Since every run of playwright test opens up a new HTTP server on the playwright-report directory (although by default the report is only served on failure), a lot of these would end up running at the same time (each on a different, probabaly randomly assinged, port but on the same underlying folder playwright-report) after tests have run enough time due to file changes detected by chokidar. Therefore, I configure the HTML reporter in playwright.config.ts to generate the HTML report but not to serve it:

reporter: [['html', { open: 'never' }]],

and instead launch http-server on the report directory only once at the time the script is launched. I do this by running the actual watch/test command in parallel with http-server using concurrently (see definition of test-playwright:watch above).

Then at the end of every run of playwright-test that had started on detection of a file change, the URL of the report is printed so you can just Ctrl+Click it to look at the report in a browser window:

To get the above scripts to work, I think the following NPM packages would need to be installed (in addition to playwright of course):

npm install -D concurrently http-server chokidar npm-run-all

Finally, for anyone using VS Code who wants test-playwright:watch script to start running automatically whenever the project folder is opened, they can define a task in .vscode/tasks.json like this (full tasks.json shown so you can just copy and paste it in a blank .vscode/tasks.json if you don't have one already). Note for this to work you would also need to configure webServer in playwright.config.ts to launch your app whenever tests start running. This is shown after tasks.json below.

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Playwright watch",
      "type": "shell", //"process" does not work as value
      "command": "npm run test-playwright:watch",
      "args": [],
      "isBackground": true, //this makes it keep running in the background without interrupting what you're doing
      "runOptions": {
        "runOn": "folderOpen" //start this task when folder is opened in VS Code
      },
      "presentation": {
        "panel": "dedicated"
      },
      "problemMatcher": {
        "fileLocation": ["relative", "${workspaceFolder}/tests"],
        "severity": "error", //probabaly redundant as it is the default
        "pattern": {
          "regexp": "^\\[TEST\\]\\s*\\[(.*)\\]\\s\\S\\s(.*):(\\d+):(\\d+).*$",
          "file": 2,
          "line": 3,
          "column": 4,
          "message": 1

        },
        "background": {
          "activeOnStart": false,
          //I have verified that if either of the following regexes is wrong,
          //you don't get the tick on the terminal to indicate once
          //a test run has completed (even though our watch task carries on running)
          "beginsPattern": "^\\[TEST\\]\\s*Running \\d* tests using \\d* workers\\s*$",
          //"beginsPattern": "^\\s*Running \\d* tests using \\d* workers\\s*$",
          //USE THE FOLLOWING endsPattern IF REVERTING TO CHOKIDAR TO WATCH
          "endsPattern": "^\\[TEST\\]\\s*To open last HTML report run:\\s*$"
          //"endsPattern": "^Waiting for file changes\\. Press enter to run tests, q to quit or h for more options\\.\\s*"
        }
      }
    }
  ]
}

In playwright.config.ts, configure webServer sort of like this:

webServer: {
    command: 'npm run dev',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
    ignoreHTTPSErrors: true,
  },

With the three patterns in problemMatcher in the task (regex in pattern block, beginsPatter and endsPattern), the VS Code background task provides a nice indication of its status in the header of the terminal window in which it is running:

If there are failures, you can see indications in a few different places:

• terminal window label (shown above)

• The terminal itself (if in view), where you can see the failing tests listed and Ctrl+Click URL of the test run report to open it in a browser (also shown above)

• In Problems pane, from where you can also navigate directly to the the test that failed:

  

[aslushnikov]

Everybody: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar?

Let us know what you think!