Getting a Power CI for kubernetes

[Pensu]

Hi all,

Since release 1.3.0-alpha3, gcr.io has all the images required to run k8s on powerpc architecture. It would be good to have a CI setup for power, so that future stuff doesn't break on it. How should I proceed with that? I have been told one way is to get the servers in CNCF, but again what would be the specs in that case? Alternatively who is the person to contact?

TIA.

cc: @luxas @bgrant0607 @roberthbailey @thockin

[roberthbailey]

You should chat with the @kubernetes/test-infra-maintainers folks.

[luxas]

First off, I'm obviously a huge +1 on this.

As the implementor of this feature, I would say that we have one important CI layer already: Does it compile on Power?

On every PR, k8s will be built for arm, arm64 and ppc64le, and e2e will fail if the code doesn't compile on the given platform (example: earlier there were +// build linux,amd64 in some places in the code, which broke all other platforms, and some other things as well I had to fix before I made the arm64 and ppc64le PR).

But although Go should produce next-to-identical code regardless of platform, bugs will sometimes show up, so it would also be good to have a nightly e2es (including windows when we get so far)

I think, however, that if we setup something in collaboration with CNCF or whoever, we should setup arm and arm64 CIs as well.

BTW, I'm writing a proposal in the meantime about Kubernetes on different platforms.
I'll cc you on it later

[luxas]

xref: #17981

[ixdy]

cc @kubernetes/sig-testing

[luxas]

@ixdy Opinions on this?

[ixdy]

I think it's a great idea, but (obviously) the challenge is getting hardware. I don't have any leads on that, but if anyone has hardware and wants to start running tests, we can definitely federate those results into our dashboards.

[bpradipt]

@ixdy if you can share min spec of the hardware required for CI (cpu, mem, disk), we can take a look and come back with availability.

[seelam]

yes, we can get the hardware access, if we get the specs.

[ixdy]

I'm not sure if I'm the best person to answer that question.

For the majority of our CI testing on GCE, we use 3-node clusters:

• master: n1-standard-1 (1 vCPU, 3.75 GB ram), 20 GB SSD PD
• nodes (3): n1-standard-2 (2 vCPU, 7.5 GB ram), 100 GB non-SSD PD

For our larger-cluster tests, we use a larger master node. "Real" clusters have different sizes, too.

Our beefiest machines (with 32 vCPU) are those running builds and unit tests, but I don't imagine we need to do builds on power - that's what crossbuild is for.

I have no idea what's typical for non-GCE deployments. And I think the arm testing @luxas has been all single-machine using Docker, so he probably has a better idea of the specs there.

[ixdy]

@kubernetes/sig-testing have any opinions about usual hardware specs you use for clusters in CI testing?

[ixdy]

I'll also note that we have many CI clusters running simultaneously, but that level of parallelism/scale isn't strictly necessary. We can scale testing to as little or as much hardware as we have. Just running the conformance subset of tests somewhere semi-regularly would be a great start.

[luxas]

A nightly build shouldn't be so heavyweight imo

I haven't run e2e tests yet on the Raspberry Pi (issue #22876) , but I think it'll handle it (especially if it's 1-5 times a day)

For arm64 @tianon recommends this one: moby/moby#19013 (comment) 😄

Of course, it's possible to run both arm and arm64 binaries on an arm64 board.
WDYT about arm CI @ixdy? That would be awesome!

But for ppc64le, we have to fix this issue first: #25886