CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow

[daniel-bartley]

Description

Unlike the operators I'm accustomed to, the default logger is not returning the cloud run jobs logs to the airflow task.

Use case/motivation

Avoids the context switching of navigating to GCP Cloud or Cloud Logging to access logs.

To submit I'd need guidance on the testing in particular. Well and the debugging of course. I don't understand why the logging is not working.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[dirrao]

@daniel-bartley You can only see the cloud-run job API calls logging to create and run the jobs. Cloud run job logs are not available in airflow as it is not running in the airflow worker.

[potiuk]

Yep. What @dirrao wrote. But If you would like to implement it as Cloud Run Operator feature to monitor, poll and stream such logs to airflow logs - it's entirely ok (as long as CloudRun API would alllow it). We are already doing similar things for example in KubernetesPodOperator, where we basically stream the logs from Pod running elsewhere and log them through airflow logging system.

[daniel-bartley]

We currently use KubernetesPodOperator for dbt.
The intuitive expectations was that the logs would be similar and we were surprised they are not.

Where in the code does KubernetesPodOperator poll and stream logs? If you can point me to in the code and provide commentary, I might be able to adjust the CR operator.

Not having the logs available in Airflow significantly diminishes airflow's fit-for-purpose for the task.

[potiuk]

We currently use KubernetesPodOperator for dbt.
The intuitive expectations was that the logs would be similar and we were surprised they are not.

Airflow is collectively created by ~ 2800 people. I am sometimes surprise the opposite, how coherent a small group of maintainers make those people (like you) to do things. And one of the reason is that they - like you notice differences and are willing to contribute back (like you) for a fantastic piece of software they can use absolutely for free and implement and improve things as partially a way to thank all those 2800 and make it even more fit-for-purpose.

Where in the code does KubernetesPodOperator poll and stream logs? If you can point me to in the code and provide commentary, I might be able to adjust the CR operator.

You will find a few classes and code here:
https://github.com/apache/airflow/blob/390eacb01a710239e69f71d0c52134b7e87d17d4/airflow/providers/cncf/kubernetes/utils/pod_manager.py

Commentary is quite straightforward. Not sure if you can do it in similar way, becuase Kuberntes POD API to read logs is quite a bit complex and there is a lot of code to provide resilience, avoid duplication and generally retrieve the logs from running POD in an efficient way, taking into account the weaknesses of the API and fact that there could be multiple containers, in the pod, each with its own logs for example.

But if I were to point you to important part of code, what is important is ONE line.

https://github.com/apache/airflow/blob/390eacb01a710239e69f71d0c52134b7e87d17d4/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L457C33-L457C89

  self.log.info("[%s] %s", container_name, message_to_log)

When yo know what next log line you should log just ... log it . The key complexity is that the task run locally from airflow side needs to retrieve the logs from Cloud Run while they are coming and log them one-by-one. Do it in a streaming way ideally so that they show up as they are coming and handle it in an efficient way.

Not having the logs available in Airflow significantly diminishes airflow's fit-for-purpose for the task.

That's why we ask you (or anyone who will grab the good first issue) to make it more fit-for-the-purpose. Generally this is how Open source software is created. Someone sees a need to improve it and ... improves it.