Skip to content

Fix CI and VRT flakiness and tech debt #1119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jul 28, 2025
Merged

Conversation

rezrah
Copy link
Collaborator

@rezrah rezrah commented Jul 26, 2025

Summary

Addresses the recent flakiness and instability in CI and VRT

List of notable changes:

  • Disables Remix E2E test as the v2 CLI is no longer maintained.
  • Updates our package-lock following last week's release
  • Visual regression tests:
    • Switch testing method from toMatchSnapshot to toHaveScreenshot as the latter is now the preferred method for screenshot comparison using Playwright.
    • Increase retries to 3
    • Max failures set, to short-circuit test suite
    • Change maxDiffPixels to maxDiffPixelRatio to prefer a percentage-based tolerance that scales better with image size. Big images previously had fase positives because maxDiffPixels value was suitable only for smaller images. Now uses a 0.02% diff threshold. This is very aggressive, but seems to be stable enough over multiple VRT runs.
  • Remove SubNav's keyboard navigation story from the VRT, as it's an interaction test only
  • Removed animated IDE from recipe stories, as they were causing flakiness

Contributor checklist:

  • All new and existing CI checks pass
  • Tests prove that the feature works and covers both happy and unhappy paths
  • Any drop in coverage, breaking changes or regressions have been documented above
  • UI Changes contain new visual snapshots (generated by adding update snapshots label to the PR)
  • All developer debugging and non-functional logging has been removed
  • Related issues have been referenced in the PR description

Reviewer checklist:

  • Check that pull request and proposed changes adhere to our contribution guidelines and code of conduct
  • Check that tests prove the feature works and covers both happy and unhappy paths
  • Check that there aren't other open Pull Requests for the same update/change

Copy link

changeset-bot bot commented Jul 26, 2025

⚠️ No Changeset found

Latest commit: ad1624e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link
Contributor

github-actions bot commented Jul 26, 2025

🟢 No design token changes found

Copy link
Contributor

github-actions bot commented Jul 26, 2025

🟢 No visual differences found

Our visual comparison tests did not find any differences in the UI.

@rezrah rezrah marked this pull request as ready for review July 28, 2025 08:47
@Copilot Copilot AI review requested due to automatic review settings July 28, 2025 08:47
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR modernizes the visual regression testing approach for improved stability and reliability by switching from the deprecated toMatchSnapshot method to the newer toHaveScreenshot method across all Playwright visual test files.

  • Switch from toMatchSnapshot to toHaveScreenshot for all visual regression tests
  • Update test ordering in PricingOptions tests by reordering "Leading Component" and "One Option" test groups
  • Maintain consistent test structure and timeout patterns across all affected files

Reviewed Changes

Copilot reviewed 37 out of 126 changed files in this pull request and generated no comments.

Show a summary per file
File Description
packages/react/src/Prose/Prose.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/PricingOptions/PricingOptions.visual.spec.ts Updated screenshot assertions and reordered test groups
packages/react/src/Pillar/Pillar.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/Pagination/Pagination.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/MonaSans/MonaSans.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/MinimalFooter/MinimalFooter.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/LogoSuite/LogoSuite.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/Link/Link.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/Label/Label.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot
packages/react/src/InlineLink/InlineLink.visual.spec.ts Updated screenshot assertions from toMatchSnapshot to toHaveScreenshot

@rezrah rezrah changed the title Update stale visual snapshots Fix CI and VRT flakiness and tech debt Jul 28, 2025
@rezrah rezrah merged commit 907d661 into main Jul 28, 2025
20 checks passed
@rezrah rezrah deleted the rezrah/fix-flakey-snapshots branch July 28, 2025 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant