Feature request: Filter or view for non-OK results to find problematic tests

[zcorpan]

I would like to make wpt.fyi be able to filter or create a view for non-OK results (harness errors, timeouts). I've looked a bit at the code today but don't know yet what to do (seems like the data is tailored for pass/total currently, and other results is visible only when looking at a file result).

This would effectively be the same as https://github.com/bocoup/wpt-error-report/ (apparently updating this to current data is not straightforward)

The benefit is that it would be easy to find tests that are problematic and need fixes either to testing infrastructure or the test themselves.

[lukebjerring]

Feature makes sense.

Some considerations:

• If filtering to a particular result (e.g. OK) but showing all browsers, what do we show when say, Chrome is OK but the other three are not? i.e. is the filter 'at least one of the browsers has this result' or 'show only results with this value'.
• And, above answer in mind, what happens with totals when filtering? Should we just omit the total entirely?
• Is filtering on a specific status enough, or should we implement this as a search extension, e.g. result:OK and -result:TIMEOUT (not timeouts)

[zcorpan]

For my ise case (finding problematic tests) I think

• at least one of the browsers has this result (otherwise I'd miss tests that e.g. fail in one browser and timeout in others)
• omitting total seems ok
• either is ok, though being able to filter for "not OK" would be nice

[foolip]

@lukebjerring, the interop view and anomalies view highlights some of the non-OK results. Do you have a staging instance with those for @zcorpan to look at?

[lukebjerring]

Do now - turns out the migration has affected the namespace of the existing data.
https://lukebjerring-results-prefix-dot-wptdashboard.appspot.com/interop/

[mdittmer]

Would it be more or less helpful to support -safari:OK rather than -status:OK ?

…

On Thu, Apr 19, 2018, 5:19 PM Luke Bjerring, ***@***.***> wrote: Do now - turns out the migration has affected the namespace of the existing data. https://lukebjerring-results-prefix-dot-wptdashboard.appspot.com/interop/ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#73 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABsWSHEEvngTxFHj3uk09o5dbI97Vm0zks5tqP9agaJpZM4TcA9M> .

[foolip]

More! If we could support filtering on the individual statuses (PASS/TIMEOUT/FAIL, harness errors) and perhaps some shorthand for "all" and n/m, that'd support most "find the problems" use cases I can think of at least. But that obviously can't be precomputed :)

[lukebjerring]

So I started playing around with this. There are currently a couple blockers:

• Search doesn't currently do anything on an individual test-file summary
• The summary format we currently use for the UI collapses the data into arrays of [passing, total]

We'd need non-lossy data conversion (which will be a side effect of creating the results-receiver endpoint - @Hexcles, something to keep in mind), and we'd need to have a search for the individual file summaries too. Filed #81 to cover that.

[Hexcles]

That summary is internal to the dashboard I think, so we have a lot of flexibility there (e.g. we might even use NoSQL/memcache instead of a summary JSON). This is essentially what I considered as "Exploring new data models" in follow-up work. @lukebjerring feel free to dump some thoughts there and we can discuss later.

[lukebjerring]

Closing as obsolete / fixed.