Improve status check handling for GKE Autopilot clusters

[briandealwis]

Can we improve the status check reporting when deploying to a GKE Autopilot cluster — informing the user that the cluster/node is being scaled up to accomodate the new job?

Waiting for deployments to stabilize...
 - deployment/leeroy-app: 0/3 nodes are available: 1 Insufficient memory, 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1623436397}, that the pod didn't tolerate, 2 Insufficient cpu.
    - pod/leeroy-app-c469448b5-wb2db: 0/3 nodes are available: 1 Insufficient memory, 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1623436397}, that the pod didn't tolerate, 2 Insufficient cpu.
 - deployment/leeroy-web: 0/3 nodes are available: 1 Insufficient memory, 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1623436397}, that the pod didn't tolerate, 2 Insufficient cpu.
    - pod/leeroy-web-99d978f66-9dr2j: 0/3 nodes are available: 1 Insufficient memory, 1 node(s) had taint {ToBeDeletedByClusterAutoscaler: 1623436397}, that the pod didn't tolerate, 2 Insufficient cpu.
[large pause]
 - deployment/leeroy-web is ready. [1/2 deployment(s) still pending]
 - deployment/leeroy-app is ready.

If the pod is not scheduled, we could look at the events to see if there was a TriggeredScaleUp event.

[tejal29]

sounds like a nice feature. Is there a way we could detect if its a autopilot cluster?

[briandealwis]

I've been told that looking at the pod events should show cluster autoscaling or node auto provisioning events. Both are briefly described here:

https://cloud.google.com/architecture/best-practices-for-running-cost-effective-kubernetes-applications-on-gke

[ValentinFunk]

Is there a way to use this with autopilot clusters yet? Deployments always fail for me (something about unscheduable), even if I see a TriggeredScaleUp