Allow passing config_path as a parameter to KubernetesExecutor

[pablo-moreno-mancebo]

Description

Currently, when using executor_config with KubernetesExecutor, the config_file path must be hardcoded. If we try to pass it dynamically using a Jinja template ("{{ params.config_path }}"), it results in an error.

Example of a working static configuration:

executor_config={"KubernetesExecutor": {"config_file": "/opt/airflow/config/kubernetes/docker-desktop-config.yaml"}}

However, if we attempt to pass it dynamically, it fails:

executor_config={"KubernetesExecutor": {"config_file": "{{ params.config_path }}"}}

This would be useful when deploying DAGs across different environments (e.g., multiple regions) without hardcoding paths.

Proposed Solution:

Modify executor_config to allow templating in KubernetesExecutor.config_file, similar to other templated parameters in Airflow.

Use case/motivation

This would be useful when deploying DAGs across different environments (e.g., multiple regions) without hardcoding paths.
Airflow should allow config_file to accept templated values, resolving them at runtime.
This feature would improve flexibility when running KubernetesExecutor tasks that need different configurations based on dynamic parameters.
Would love to hear thoughts from the community on this! Thanks! 😊

Related issues

This feature would improve Airflow’s flexibility in hybrid execution environments and align with the goals of AIP-61. Would love to hear thoughts from the community on this! 🚀

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

If you can take a look at propose a PR solving it, yes I think it would be useful. But it needs a deep look in the code whether it's possible, also with Airflow 3 changes / Task SDK changes in progress, likely doing it after this is completed, is a good idea.

[insomnes]

Are you sure you can rewrite it with?

executor_config={"KubernetesExecutor": {"config_file": "/opt/airflow/config/kubernetes/docker-desktop-config.yaml"}}

I've dug through the code and also experimented a bit, and couldn't find any way to rewrite the initial config file for KubernetesExecutor. Am I missing something? Any of the executor_config I've tried to use with deprecated KubernetesExecutor variant except the empty one led to the error:

{kubernetes_executor.py:355} ERROR - Invalid executor_config for TaskInstanceKey

[Maryam3107]

Description

Currently, when using executor_config with KubernetesExecutor, the config_file path must be hardcoded. If we try to pass it dynamically using a Jinja template ("{{ params.config_path }}"), it results in an error.

Example of a working static configuration:

executor_config={"KubernetesExecutor": {"config_file": "/opt/airflow/config/kubernetes/docker-desktop-config.yaml"}}
However, if we attempt to pass it dynamically, it fails:

executor_config={"KubernetesExecutor": {"config_file": "{{ params.config_path }}"}}
This would be useful when deploying DAGs across different environments (e.g., multiple regions) without hardcoding paths.

Proposed Solution:

Modify executor_config to allow templating in KubernetesExecutor.config_file, similar to other templated parameters in Airflow.

Use case/motivation

This would be useful when deploying DAGs across different environments (e.g., multiple regions) without hardcoding paths. Airflow should allow config_file to accept templated values, resolving them at runtime. This feature would improve flexibility when running KubernetesExecutor tasks that need different configurations based on dynamic parameters. Would love to hear thoughts from the community on this! Thanks! 😊

Related issues

This feature would improve Airflow’s flexibility in hybrid execution environments and align with the goals of AIP-61. Would love to hear thoughts from the community on this! 🚀

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

I can solve this problem. kindly assign it to me

[Nataneljpwd]

Hello, could anyone clarify further on what needs to be done?
How will an end result dag look like?
I'm looking into it and this setting is possible on task and dag level, which one should be implemented? Isn't it better to implement it for the task instead?