Kaniko multi-platform build actually runs as single-platform

[rayshoo]

This issue was discovered while working on PR #9860.
The tests were conducted using the official Skaffold image:
skaffold:latest (gcr.io/k8s-skaffold/skaffold:latest).

When building with Kaniko through Skaffold, the platforms array is effectively locked to the first index only, due to the nodeSelector (kubernetes.io/arch) being fixed. As a result, subsequent platforms are not built for their intended architecture but instead fall back to the first platform.

For example, with the following configuration:

platforms:
  - linux/amd64
  - linux/arm64

After the first linux/amd64 build, the second build should be scheduled on a linux/arm64 node. However, it is still scheduled on the linux/amd64 node (nodeSelector: kubernetes.io/arch=amd64) and thus produces another amd64 image. The image is tagged and pushed as IMAGE_NAME:IMAGE_TAG_arm_64, and grouped together as if it were a valid multi-platform image, but in reality the supposed arm64 image is also recognized as amd64.
On the other hand, if linux/arm64 is the first index in the platforms array, all remaining images are also built as arm64.

It seems this might be fixed by adjusting the nodeSelector, but the codebase is quite large and difficult for me to patch alone. I’m filing this issue in case anyone knows a solution or can help address it.

# skaffold.yaml used for testing
---
apiVersion: skaffold/v4beta13
kind: Config
build:
  tagPolicy:
    envTemplate:
      template: multi-arch-test
  artifacts:
  - image: harbor.deleo.co.kr/chan/test
    context: .
    platforms:
    - linux/amd64
    - linux/arm64
    kaniko:
      cleanup: true
      useNewRun: true
      snapshotMode: redo
  cluster:
    namespace: test
    dockerConfig:
      secretName: test
    tolerations:
    - effect: NoSchedule
      key: karpenter/node.build
      operator: Exists
    nodeSelector:
      node: build
    resources: {}

ubuntu@docker-0:~/test/skaffold$ docker pull harbor.deleo.co.kr/chan/test:test-2-origin
test-2-origin: Pulling from chan/test
6e174226ea69: Pull complete
4a821b521af1: Pull complete
Digest: sha256:651a8739815fcd8dabf13964f80e3ce3901ca554c38e616e3d58b65708629ff3
Status: Downloaded newer image for harbor.deleo.co.kr/chan/test:test-2-origin
harbor.deleo.co.kr/chan/test:test-2-origin

ubuntu@docker-0:~/test/skaffold$ docker run harbor.deleo.co.kr/chan/test:test-2-origin
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested
exec /app: exec format error

ubuntu@docker-0:~/test/skaffold$ docker pull harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64
test-2-origin_linux_amd64: Pulling from chan/test
Digest: sha256:0dabff50c76c53423584b083e7d7a91ec627a9c516938c46b045af35811961ab
Status: Downloaded newer image for harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64
harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64

ubuntu@docker-0:~/test/skaffold$ docker run harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested
exec /app: exec format error

ubuntu@docker-0:~/test/skaffold$ docker pull --platform amd64 harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64
test-2-origin_linux_amd64: Pulling from chan/test
Digest: sha256:0dabff50c76c53423584b083e7d7a91ec627a9c516938c46b045af35811961ab
Status: Image is up to date for harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64
harbor.deleo.co.kr/chan/test:test-2-origin_linux_amd64

ubuntu@docker-0:~/test/skaffold$ docker run --platform amd64 harbor.deleo.co.kr/chan/test:test-2-origin
exec /app: exec format error