GitDagBundle: DAG processor fails to load versioned DAGs due to missing materialized folders

[liyude-tw]

Apache Airflow Provider(s)

git

Versions of Apache Airflow Providers

No response

Apache Airflow version

3.0.2

Operating System

Ubuntu 20.04.6 LTS

Deployment

Virtualenv installation

Deployment details

No response

What happened

When using GitDagBundle with supports_versioning=True, Airflow DAG Processor fails to execute DAG callbacks because it cannot find the materialized code under versions/<commit_hash>/.

Only the tracking_repo/ folder is updated with the latest version, but when a DAG-level callback (e.g. on_success_callback, on_failure_callback) is triggered, the callback request sent to the DAG Processor points to the versioned folder (e.g. /dag-bundles/git_xxx/versions/<commit_hash>/dag.py), which does not exist by default.

This leads to a failure in the DAG Processor when trying to load the DAG from the file. Specifically, inside DagFileProcessor._execute_dag_callbacks, the following code fails:
dag = dagbag.dags[request.dag_id] # returns empty

This happens because dagbag.process_file() cannot find the DAG file at the specified path (versions/<commit_hash>/...), and so the DAG is never loaded, and the callback is silently skipped.

What you think should happen instead

The DAG Processor should also materialize the correct version of the DAG source code (i.e., clone the Git repository to the appropriate versions/<commit_hash> path) before attempting to load and execute DAG-level callbacks.

How to reproduce

1. Use GitDagBundle with supports_versioning=True.
2. Define a DAG-level on_failure_callback or on_success_callback.
3. Deploy a new Git version and trigger a DAG run that invokes the callback.
4. The DAG tasks execute successfully.
5. But DAG callback fails because the path in the callback request is under versions/<commit_hash>/, which is not materialized
6. As a workaround, we patched GitDagBundle.refresh() to also clone the repo into the versions/ directory.

Anything else

In our environment, DAGs are triggered via Asset, but this should not affect the behavior.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[ephraimbuddy]

Hi @liyude-tw, can I see the error you encountered here?

[LucasRoesler]

I believe I have experienced the same issue, currently using Airflow 3.0.6 with versioned DAG Bundles

The DAG level callbacks never execute and the only information is this in the DAG Processor.

--- Supervised process Last chance exception handler ---
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 387, in _fork_main
    target()
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 172, in _parse_file_entrypoint
    result = _parse_file(msg, log)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 188, in _parse_file
    _execute_callbacks(bag, msg.callback_requests, log)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 230, in _execute_callbacks
    _execute_dag_callbacks(dagbag, request, log)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 236, in _execute_dag_callbacks
    dag = dagbag.dags[request.dag_id]
          ~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'email_alert_test_dag'

[nailo2c]

Hi, is this issue still exist? I can't reproduce it from my local device on the current main branch.

Here is my setting:

1. DAG

from airflow import DAG
from airflow.operators.bash import BashOperator
import logging

def _on_success(context):
    logging.getLogger().warning(">>> SUCCESS CALLBACK TRIGGERED <<<")

default_args = {
    "owner": "airflow",
    "retries": 0,
}

with DAG(
    dag_id="git_bundle_success_demo",
    default_args=default_args,
    schedule=None,
    on_success_callback=_on_success,
    catchup=False,
) as dag:
    BashOperator(
        task_id="always_succeed",
        bash_command="echo 'simulating success' && exit 0",
    )

2. Config

dag_bundle_config_list = [{
    "name": "test_git_bundle",
    "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
    "kwargs": {
        "repo_url": "https://github.com/nailo2c/my_repo.git",
        "tracking_ref": "main"
    }
}]

After I ran it, I got logs like the ones below:

root@3b9ea3cba03a:/opt/airflow# cat ~/airflow/logs/dag_processor/latest/test_git_bundle/example_git_bundle_success_dag.py.log | grep "SUCCESS CALLBACK TRIGGERED"
{"timestamp":"2025-10-26T23:22:00.320640Z","level":"info","event":">>> SUCCESS CALLBACK TRIGGERED <<<","logger":"task.stdout"}
{"timestamp":"2025-10-26T23:26:19.995804Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":6}
{"timestamp":"2025-10-26T23:36:36.206576Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":7}

Then I committed a change to update the log to >>> SUCCESS CALLBACK TRIGGERED VERSION 2 <<<. After waiting for the DAG Processor to refresh it, I got logs like the ones below:

{"timestamp":"2025-10-26T23:22:00.320640Z","level":"info","event":">>> SUCCESS CALLBACK TRIGGERED <<<","logger":"task.stdout"}
{"timestamp":"2025-10-26T23:26:19.995804Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":6}
{"timestamp":"2025-10-26T23:36:36.206576Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":7}
{"timestamp":"2025-10-26T23:53:06.247669Z","level":"warning","event":">>> SUCCESS CALLBACK TRIGGERED VERSION 2 <<<","logger":"root","filename":"example_git_bundle_success_dag.py","lineno":7}

It seems to be working well on my side.