Screen record system tests

[seanbudd]

Related issues, PRs or discussions

Related #18597

What is the current state of the codebase?

Our system test suite takes a screenshot if any system test fails.
This helps add some context for sighted developers, but not much.

Why are changes required?

A full video of the system test would add more context on what's causing the failure.
Examples:

• a popup window stealing focus
• changing to the wrong window
• a key being mishandled. Real example: nvda+t being handled as t causing the notepad system test to edit the file.

What technical changes are required?

Use Win+alt+r (Windows Game bar feature) to record each system test.
Ensure each system test has log viewer and speech viewer enabled so speech and logs can be seen real time.
Keep the window size minimal and place the dialogs appropriately to minimize interference with tests.
Upload video on failure.

Are the proposed technical changes API breaking?

No

Are there potential risks or issues with the proposed implementation?

Unfortunately there is no audio on these machines so this would only be useful to sighted devs

[gexgd0419]

Can we install a virtual audio device before testing to get the audio work?

actions/runner-images#6983

GitHub is not willing to include one by default, but maybe we can do this ourselves.

Another related issue: actions/runner-images#2528

[seanbudd]

@gexgd0419 - that would be helpful if it can easily be done. Do you have any suggestions?

[gexgd0419]

We can check the issues mentioned above, where they tried to install an audio driver called scream, and provided the workflow files they used. We can experiment with them first.

[seanbudd]

Scream seems no longer actively maintained and untested on Windows 11 - duncanthrax/scream#227

[seanbudd]

This seems promising: https://github.com/VirtualDrivers/Virtual-Audio-Driver

[seanbudd]

Never mind - a lot of issues with recent release: https://github.com/VirtualDrivers/Virtual-Audio-Driver/issues

[gexgd0419]

I have used a driver called VB-Audio Virtual Cable, which creates a pair of audio input and output device, and passes the audio from the input to the output. It's a donationware. Not sure whether it's suitable here though.

[gexgd0419]

Also note that Win+Alt+R only records one window -- the active window when recording starts, not the entire desktop. Then it will always record the same window whether it is active or not, and closing the window or changing the window size during recording will stop recording immediately. If you want to record the whole screen, you'll have to use something else.

If the environment is Windows 11 and has the latest built-in app versions, the Snipping tool (Win+Shift+S) can be used instead to record the screen. Not that easy to automate though, as you would have to draw a rectangle area for recording with the mouse.

[seanbudd]

Ah - we want to record the full desktop. Win+shift+s is probably the way to go

[gexgd0419]

I think that the Snipping tool is not easy to automate.

You can use Win+Shift+R instead of Win+Shift+S to start Snipping tool in recording mode. But still, you need to specify a recording area, either by dragging the mouse, or by using keyboard - yes, using keyboard to do that is possible, by tabbing into the screen area, using arrow keys to move the cursor (holding Ctrl to move faster), and using Enter/Space to select the corner points.

Snipping tool also has no global shortcut keys for starting, pausing, or stopping recording. You will have to either use mouse to click the buttons, or Alt+Tab to the Snipping toolbar and select the button.

So maybe it's better to experiment with some other tools that can be automated more easily first. For example, capture Windows screen with FFmpeg.