-
Notifications
You must be signed in to change notification settings - Fork 0
Create Playwright Summary
Important
All of the files in this section should be created in the gh-pages
branch.
Add the following file structure. The automated scripts and GitHub Actions assume this file structure, so they would need to be updated accordingly if the structure is changed.
|-- _data
|-- playwright-reports
|-- branches
|-- _layouts
|-- playwright-reports
|-- branches
Note
The standard JSON Playwright report has this format.
{
"config": {...},
"suites": [...],
"errors": [],
"stats": {
"startTime": "2025-02-25T00:28:32.978Z",
"duration": 154381.398,
"expected": 82,
"skipped": 2,
"unexpected": 0,
"flaky": 0
}
}
This JSON does not indicate the total number of tests and the overall results of the tests.
The total number of tests can be calculated by adding stats.expected
(passing tests), stats.unexpected
(failing tests), stats.skipped
(tests that were skipped), and stats.flaky
(tests that were flaky). If there are errors in the tests, the stats for expected
, unexpected
, skipped
, and flaky
will all be 0.
The overall results of the tests can be determined by checking if stats.unexpected
is greater than 0, because that means 1 or more tests failed, therefore the overall result is failed. If there are errors in errors
, then the overall result is a warning. Otherwise, the overall result is passing.
We can use a filter plugin to calculate these values and reformat the data to make it easier to use.
Create _plugins/playwright-results-filter.rb
. The helper method playwright_status
uses the JSON to determine if the tests passed, failed, or errored out.
module Jekyll
module PlaywrightResultFilter
def normalize_playwright(input)
status = playwright_status(input)
total = playwright_total(input)
stats = input["stats"]
stats["errors"] = input["errors"]
stats["total"] = total
stats["status"] = status
stats
end
def playwright_total(input)
if input["errors"].size > 0
0
else
input["stats"]["expected"] + input["stats"]["unexpected"] + input["stats"]["skipped"]
end
end
def playwright_status(input)
if input["stats"]["unexpected"] > 0
# if the unexpected results is greater than one, return fail
"fail"
elsif input["errors"].size == 0
# if there are no unexpected results and no errors, return pass
"pass"
else
# otherwise if there are errors, return error
"warn"
end
end
end
end
Liquid::Template.register_filter(Jekyll::PlaywrightResultFilter)
Create a file _includes/playwright-summary.html
to display the data from the json report. This is an example of what we are using.
Tip
_includes/playwright-summary.html
This snippet expects values to be passed in using parameters data
(JSON Playwright report data), branch
(the branch the report ran on), run_id
(the id of the GitHub Action run), and url
(the url to the static HTML Playwright report).
{% if include.data %}
<!-- use the custom normalize_playwright filter to get the status, total number of tests, errors, and test statistics from the data -->
{% assign results = include.data | normalize_playwright %}
<!-- if the data was included, display the summary -->
<div class="test-report-container">
<div class="report-header">
<!-- if the run_id was included, display it in the header of the summary -->
{% if include.run_id %}
<!-- capture also creates strings with variables, create the url back to the run -->
{% capture run_link %}{{ site.github_url }}/actions/runs/{{ include.run_id }}{% endcapture %}
<!-- no relative_url necessary here because the link it outside of the app -->
<span class="report-run">Run Id: <a href="{{ run_link }}" target="_blank">{{ include.run_id }}</a></span>
{% else %}
<span class="report-run"></span>
{% endif %}
<!-- Liquid provides a helper method for formatting dates -->
<span class="report-time">Run At: {{ results.startTime | date: "%Y-%m-%d %H:%M %Z" }}</span>
</div>
<div class="report-body">
<!-- style the icon according to the report status and use the custom status_icon filter to return the correct svg -->
<span class="status-icon status-{{results.status}}">{{ results.status | status_icon }}</span>
<div class="results">
{% if results.status == "error" %}
<!-- if there are errors, there won't be any test statistics, so only display the errors -->
<div class="report-errors">
{% for error in results.errors %}
<!-- loop through the errors and display the messages -->
<div class="report-error">{{ error.message }}</div>
{% endfor %}
</div>
{% else %}
<!-- if there are no errors, display the test statistics -->
<div class="report-stats">
<div class="stat-item">
All <span class="stat-count">{{ results.total }}</span>
</div>
<div class="stat-item">
Passed <span class="stat-count">{{ results.expected }}</span>
</div>
<div class="stat-item">
Failed <span class="stat-count">{{ results.unexpected }}</span>
</div>
<div class="stat-item">
Flaky <span class="stat-count">{{ results.flaky }}</span>
</div>
<div class="stat-item">
Skipped <span class="stat-count">{{ results.skipped }}</span>
</div>
</div>
{% endif %}
<!-- Liquid provides a helper method that will create the link relative to the baseurl set in _config.yml -->
<span class="report-link"><a href="{{ include.url | relative_url }}" target="_blank">View Full Report</a></span>
</div>
</div>
</div>
{% else %}
<!-- if the data was not included, display a message that there is nothing to display -->
<div>There is currently no report for {{ include.branch }}</div>
{% endif %}
This snippet can be used within other pages or snippets using the include
tag.
{% include playwright-summary.html branch="branch-name" run_id="0123456789" data=data url="/playwright-reports/branches/branch-name/0123456789.html" %}
Create a custom CSS file for styling your reports. This may also be done as part of the site CSS file. This is an example of what we are using.
Tip
assets/css/playwright.css
.test-report-container {
display: flex;
flex-direction: column;
}
.report-header {
margin-top: 1rem;
border: 1px solid #d0d7de;
border-top-left-radius: 6px;
border-top-right-radius: 6px;
border-bottom: none;
background-color: #f6f8fa;
padding: 0.5rem 1rem;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
display: flex;
justify-content: space-between;
font-weight: 300;
font-size: smaller;
color: #6e7781;
}
.report-body {
border: 1px solid #d0d7de;
border-bottom-left-radius: 6px;
border-bottom-right-radius: 6px;
padding: 1rem 0 0.5rem 0;
line-height: 1.5rem;
font-size: smaller;
overflow: hidden;
text-overflow: ellipsis;
display: flex;
align-items: center;
justify-content: space-around;
}
.results {
display: flex;
flex-direction: column;
padding: 0.5rem 0;
}
.report-errors {
display: flex;
flex-direction: column;
gap: 1rem;
width: 35rem;
font-family: monospace;
}
.report-error {
border: 1px solid #d0d7de;
border-radius: 6px;
padding: 0.5rem 1rem;
}
.stat-item:first-child {
border-top-left-radius: 6px;
border-bottom-left-radius: 6px;
}
.stat-item:last-child {
border-top-right-radius: 6px;
border-bottom-right-radius: 6px;
}
.stat-item {
flex: none;
position: relative;
float: left;
padding: 0.5rem 1rem;
color: #24292f;
border: 1px solid #d0d7de;
}
.report-stats {
display: flex;
flex-direction: row;
font-weight: 500;
line-height: 2rem;
}
.report-link {
width: 100%;
display: flex;
justify-content: end;
min-height: 1rem;
}
.stat-count {
margin-left: 0.5rem;
display: inline;
min-width: 2rem;
padding: 0 0.5rem;
color: #24292f;
text-align: center;
background-color: rgba(175, 184, 193, 0.2);
border: 1px solid transparent;
border-radius: 2em;
}
.status-icon {
height: 3.5rem;
width: 3.5rem;
}
.status-pass {
color: rgb(2, 119, 2);
}
.status-fail {
color: rgb(155, 0, 0);
}
.status-warn {
color: rgb(243, 186, 17);
}
The site is not yet configured to use the styling yet, that will be added in the next section. With our snippet and styling, our Playwright report summary looks like this.