Skip to content

ExternalTaskSensor doesn't check existence in deferrable model #40745

@armandduijn

Description

@armandduijn

Apache Airflow version

2.9.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The ExternalTaskSensor in deferrable mode doesn't throw an error if the external DAG doesn't exist. Setting check_existence=True doesn't have an effect.

What you think should happen instead?

The task should immediately fail instead of staying in the deferred state.

How to reproduce

The code below implements a DAG with two ExternalTaskSensor tasks. One task is set to be deferred and the other one is old non-deferred way.

When running the DAG, the regular_sensor task will immediately retry and eventually fail, while the defer_sensor task will remain deferred.

from datetime import datetime

from airflow import DAG
from airflow.sensors.external_task import ExternalTaskSensor

with DAG(
    "Test",
    default_args={ "start_date": datetime(2024, 7, 11) }
) as dag:
    ExternalTaskSensor(
        task_id="defer_sensor",
        external_dag_id="FakeDAG",
        check_existence=True,
        deferrable=True,
    )

    ExternalTaskSensor(
        task_id="regular_sensor",
        external_dag_id="FakeDAG",
        check_existence=True,
        deferrable=False,
    )

Operating System

Windows

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions