Pod name incorrect in RenderedTaskInstanceFields

[hterik]

Apache Airflow version

2.5.0

What happened

1. Start a task running with KubernetesExecutor
2. Open the /rendered-k8s?dag_id=...&task_id=... endpoint in the web UI
3. Copy the metadata.name
4. kubectl logs pod/$name
5. Error from server (NotFound): pods "example-sla-dag-sleep-20-1bc7f8d6892849eaac4642c4177e2eab" not found

By inspecting the

What you think should happen instead

Pod name presented in Rendered K8s Pod Spec should match the pod name used for running task. Otherwise pod name should not be presented at all.

This is not only a problem for users of the UI, also in the code it can be useful to have the real pod-id.
I'm in the process of troubleshooting strange errors with task adoption, where there are frequently orphaned tasks found. Currently there appears to be no way to go from TI to pod_id, the adoption-process instead goes the other way, using pod labels to find TIs. So for any TIs that were not found in the adoption, one can not find if they at any point have had matching pods or not.

How to reproduce

No response

Operating System

Ubuntu 22.04

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

Could it be that you can't find the pod because it was deleted when reached final state? Did you set is_delete_operator_pod=False?

[hterik]

The pod executing the task is not deleted, it is visible in kubernetes but with a different name.

Pod creation happens in the scheduler, kubernetes_executor.run_pod_async. Here the real name is generated that is sent to kubernetes.

However, if i understand the flow correctly, the RenderedTaskInstanceFields is later created inside the pod itself, via the local executor when it calls taskinstance._execute_task_with_callbacks

            rtif = RenderedTaskInstanceFields(ti=self, render_templates=False)
            RenderedTaskInstanceFields.write(rtif)
            RenderedTaskInstanceFields.delete_old_records(self.task_id, self.dag_id)

RenderedTaskInstanceFields calls ti.render_k8s_pod_yaml() which calls construct_pod, where the pod name is regenerated using make_unique_pod_id, which as the name suggests is random and a new unique value :)

[eladkal]

So it seems that this effects KubernetesExecutor but not using KPO as stand alone
cc @jedcunningham @dstandish

[csp33]

I found out that the correct pod name is shown under Task Instance Details -> Hostname.
BTW, issue also affects Airflow 2.4.3

[eladkal]

ah right. so this is not a bug.. maybe we just need to clarify what pod is shown in render tab for users of Kubernetes executor

[eladkal]

@csp33 would you like to raise PR ?