-
Notifications
You must be signed in to change notification settings - Fork 15.8k
Open
Labels
affected_version:2.9Issues Reported for 2.9Issues Reported for 2.9area:async-operatorsAIP-40: Deferrable ("Async") OperatorsAIP-40: Deferrable ("Async") Operatorsarea:corecontributors-workshopIssues that are good for first-time contributor's workshopIssues that are good for first-time contributor's workshopgood first issuekind:bugThis is a clearly a bugThis is a clearly a bug
Description
Apache Airflow version
2.9.2
If "Other Airflow 2 version" selected, which one?
No response
What happened?
The ExternalTaskSensor in deferrable mode doesn't throw an error if the external DAG doesn't exist. Setting check_existence=True
doesn't have an effect.
What you think should happen instead?
The task should immediately fail instead of staying in the deferred state.
How to reproduce
The code below implements a DAG with two ExternalTaskSensor tasks. One task is set to be deferred and the other one is old non-deferred way.
When running the DAG, the regular_sensor
task will immediately retry and eventually fail, while the defer_sensor
task will remain deferred.
from datetime import datetime
from airflow import DAG
from airflow.sensors.external_task import ExternalTaskSensor
with DAG(
"Test",
default_args={ "start_date": datetime(2024, 7, 11) }
) as dag:
ExternalTaskSensor(
task_id="defer_sensor",
external_dag_id="FakeDAG",
check_existence=True,
deferrable=True,
)
ExternalTaskSensor(
task_id="regular_sensor",
external_dag_id="FakeDAG",
check_existence=True,
deferrable=False,
)
Operating System
Windows
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Metadata
Metadata
Assignees
Labels
affected_version:2.9Issues Reported for 2.9Issues Reported for 2.9area:async-operatorsAIP-40: Deferrable ("Async") OperatorsAIP-40: Deferrable ("Async") Operatorsarea:corecontributors-workshopIssues that are good for first-time contributor's workshopIssues that are good for first-time contributor's workshopgood first issuekind:bugThis is a clearly a bugThis is a clearly a bug