CI management for Tier 2 platforms

[joyeecheung]

Spinning from the #1663, during the TSC discussions today we noticed that the tiers are documented as follows in BUILDING.md

Tier 1: These platforms represent the majority of Node.js users. The Node.js Build Working Group maintains infrastructure for full test coverage. Test failures on tier 1 platforms will block releases.
Tier 2: These platforms represent smaller segments of the Node.js user base. The Node.js Build Working Group maintains infrastructure for full test coverage. Test failures on tier 2 platforms will block releases. Infrastructure issues may delay the release of binaries for these platforms.
Experimental: May not compile or test suite may not pass. The core team does not create releases for these platforms. Test failures on experimental platforms do not block releases. Contributions to improve support for these platforms are welcome.

Which does not say anything about Tier 2 platforms must be in the pull request testing CI, and there isn't much distinction between Tier 1 and Tier 2 if they are both in the testing CI.

Personally I think given the current situation of the testing CI, we are biting more than we can chew if we still insist in keeping the Tier 2 platforms in every pull request testing CI run. It requires more resources than we currently have, increase CI run time, and creates a bigger flake surface. (for example, in the reliability report yesterday you can see several tier 2 platforms on the list). It also shifts the burden of making every changes work on that platform to individual contributors, who can be new to the project and it can be frustrating for them to get blocked by these niche platforms that they don't care too much about, and overall making it harder to gain and retain new contributors.

Is it really more important to keep every pull request to pass all the tests in these tier 2 platforms than providing a frictionless experience for contributors? Personally I don't think that's the right prioritization. Keeping the tests green on a platform in the CI, specifically, is not the same as just "keeping Node.js work on that platform" - for example a test could pass perfectly fine on a well-maintained version of the platform, but still is in trouble for some reason in the CI on that platform, which may be strictly related to the state of the machine in the CI, or the specific setup of that machine. It does not seem right say for a new contributor to open a pull request to add util.something which is a straight-forward JS thing with a straight-forward test, and they end up scratching their head at some mysterious flake/machine problem on a niche tier 2 platform in the CI, and have to keep pinging other collaborators to restart the CI to get it landed (assuming they don't get frustrated enough themselves and give up on the PR). This can happen to tier 1 platform too, but at least the user base of tier 1 platforms is enough to deserve the attention of innocent new contributors, but tier 2 really isn't.

I think a more tenable solution would be to keep tier 2 platforms in the node-daily-master CI, and let Jenkins notify corresponding stakeholders when tests fail there. The stakeholders could come back opening an issue or opening a pull request to fix the test for that platform, and make sure that things are in order for the releases (if there are infrastructure issues that makes it hard to build the binaries in time for the release then they'll be delayed - which is precisely what the Tier 2 description says). For node-test-pull-request, we can either remove the tier 2 platforms, or only make sure that they can build but do not actually run the tests, as the marginal cost of keeping all the tests green on all these platforms isn't really worth the benefit at this point IMO.

cc @nodejs/tsc @nodejs/build

[targos]

I think we should rework all the Tiers to have, in a nutshell:

• Tier 1 (Maintained by the project / collaborators):
  • Tested on every pull request
  • We produce release binaries
  • Test failures prevent pull requests from landing
• Tier 2 (Maintained by the community / interested parties):
  • Tested on every pull request
  • We do not produce release binaries
  • Test failures do not prevent pull requests from landing
• Tier 3 (Experimental):
  • No tested on every pull request
• We only accept pull requests that target tiered platforms
• For Tier 2 platforms, we document who (GitHub team, individuals) responsible for their support and setup automation so they are notified when the build is broken or tests are failing.

[legendecas]

Tier 2:

• Tested on every pull request
• Test failures do not prevent pull requests from landing

These two conditions conbimed do not sound like a healthy CI situation. If the test failures do not prevent pull requests from landing, it should not be tested on every pull request. It adds consufions.

[aduh95]

I agree, CI failure of a Tier 2 platform should block landing stuff until the team responsible for that platform has been pinged and come back with a path forward (e.g. wait for a fix to land, or skip the failing test), with a maximum waiting time after which that platform would automatically be downgraded to Tier 3, and the PR can land.

[gireeshpunathil]

the stated (and desired, IMO) difference between t1 and t2 is the size of user base. not releasing binaries for t2 will affect all its users, whatever volume of users it might be in comparison with t1 users. I am worries about that approach, for the sake of providing frictionless experience to contributors.

lets compare this with security scenario. if there are less folks to fix vulnerabilities, will we let the code bleed out, or stop supporting part of the code? no, we raise funds to address it. I think for t2 platforms that have users but less contributors, we could adopt a similar mechanism, until we see the platform is not in use.

[targos]

What I imagine is that CI failures would not be as visible as Tier 1 (for example not as a Red Cross in GitHub UI)

[marco-ippolito]

I agree, CI failure of a Tier 2 platform should block landing stuff until the team responsible for that platform has been pinged and come back with a path forward (e.g. wait for a fix to land, or skip the failing test), with a maximum waiting time after which that platform would automatically be downgraded to Tier 3, and the PR can land.

I totally agree here, we should document also how/why/when a platform drops or climbs the tierlist

[targos]

@gireeshpunathil I'm not suggesting we stop releasing binaries that we already release today. If we have Tier 2 platforms for which we release binaries, they should be promoted to Tier 1

[targos]

@aduh95 what you suggest sgtm too

[joyeecheung]

I agree, CI failure of a Tier 2 platform should block landing stuff until the team responsible for that platform has been pinged and come back with a path forward (e.g. wait for a fix to land, or skip the failing test), with a maximum waiting time after which that platform would automatically be downgraded to Tier 3, and the PR can land.

I think in the short term that sounds good but we also need to watch out for the case where "nobody has the time to ping anyone and things just get stalled for no good reason", since when a PR is opened, no one is obliged to review it or look into the CI failure, this means it still falls on the individual contributors to relentlessly ping other collaborators, who might know how this work, provided that

1. The new contributor knows that this is what they need to do, not everyone gets that your PR don't get automatic sheperding and you need to take the initiative to bug other people
2. The collaborator they find do know how this works - we have many collaborators that are just casual contributors and are not up-to-date with whatever policy there is - and have the time/keep it on their radar enough to help them ping the platform teams.

And as this is a volunteer-based project, I often see PRs getting abandoned or stalled just because "the CI doesn't pass on the first run" (sometimes by collaborators, too), which is something we should make some effort to avoid or it makes it difficult for the project to gain/retain new contributors.

We can see how this pans out and if it still ends up stalling pull request that would otherwise not have this friction if the tier 2 platforms aren't blocking any PR, we should consider removing them from the PR CI altogether.

[bnoordhuis]

I think a more tenable solution would be to keep tier 2 platforms in the node-daily-master CI, and let Jenkins notify corresponding stakeholders when tests fail there. The stakeholders could come back opening an issue or opening a pull request to fix the test for that platform [..]

I wholeheartedly agree with this. Pull requests have waaaaay too much friction now, so much so that many people aren't going to bother, myself included. Take away the roadblocks and let the tier 2 platforms look after themselves.

[gireeshpunathil]

while it is reasonable in an OSS project where contributor's time is voluntary, prioritising decisions based on contributor's perspective alone may risk alienating tier2 users - "come fix it yourself if you care". the project is sufficiently matured and being used heavily in the field (enterprise and non-enterprise) and TSC should try everything it can before amending release decisions and platform support matrix:

• encouraging collaboration for those platforms
• emphasising user impact and stakeholder needs
• acknowledge t2 users may not have the expertise to maintain node.js
• add elevation of t2 platform as project's long-term vision
• outline ways to make contributions easier
  • mentoring programs
  • documentation
  • funding initiatives

this positions our relationship between contributors and users as a mutual one and reflects the spirit of community.

[joyeecheung]

prioritising decisions based on contributor's perspective alone may risk alienating tier2 users

For tier 1 users, I think that's a risk we should consider, for tier 2 I don't think so as by definition they don't have as many users, and usually they are backed by some company who answers to these tier 2 users as their own customers. By caring too much about tier 2 users, we also risk giving the community the wrong impression that they are somehow a customer to the Node.js as well and are entitled to customer service from the project (this is already not true for tier 1 users, but at least tier 1 users are less likely to pay any company in a way that can create a delusion that the money is proxied to a random Node.js project contributor that they are talking to).

"come fix it yourself if you care".

I think this is already the reality, and is how a volunteer-based project should operate in general. For actual, generic bugs that affect all platforms, it has always been "come fix it yourself if you care". There are still many bugs lying around among the 1600+ issues with dozens of thumbs up over the years while no one makes enough effort to fix them. IMO it's too much of a luxury to prioritize anything about tier 2 platforms while things are already bad enough for generic bugs on all platforms and we have been in survival mode for a long time.

TSC should try everything it can before amending release decisions and platform support matrix:

The TSC is also based on volunteers. Unless there are volunteers in the TSC actively pledged to spend the time and make this list happen, the more likely outcome is that nothing is going to be done along these lines. Shifting the "biting off more than we can chew" problem from individual contributors to a subset of them (the TSC) isn't really going to solve the problem, either, the most practical solution remains to stop taking on more responsibility than what the volunteers are able to afford.

[anonrig]

Take away the roadblocks and let the tier 2 platforms look after themselves.

I agree with @bnoordhuis here.

[richardlau]

Personally I think given the current situation of the testing CI, we are biting more than we can chew if we still insist in keeping the Tier 2 platforms in every pull request testing CI run. It requires more resources than we currently have, increase CI run time, and creates a bigger flake surface. (for example, in the reliability report yesterday you can see several tier 2 platforms on the list). It also shifts the burden of making every changes work on that platform to individual contributors, who can be new to the project and it can be frustrating for them to get blocked by these niche platforms that they don't care too much about, and overall making it harder to gain and retain new contributors.

Looking at that list there is only one entry that only appeared on a tier 2 platform. Everything else also occurred on a tier 1.