job table missing `dag_id` and `executor_class` for BackfillJob type

[Jie211]

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When using the airflow dags backfill command to execute a backfill for any DAG, I noticed that the dag_id and executor_class columns in the job table are empty(null).

Results in entries in the job table like this:

124460, NULL, running, BackfillJob, 2024-06-05 01:03:40.936253+00, NULL, 2024-06-05 01:22:41.634367+00, NULL, hoge-5566755f9-wbw57, airflow

However, in versions before 2.6.0, the entries in the job table included the dag_id and executor_class as expected:

2, bash_operator, running, BackfillJob, 2024-06-05 02:25:32.905082+00, NULL, 2024-06-05 02:25:48.017582+00, CeleryExecutor, a5dec317208c, airflow

I'm not sure exactly where the issue lies (or if this is a feature rather than a bug), but I will take a closer look at the code later. If I find the problem, I might submit a PR with a fix.

What you think should happen instead?

I'm not sure, but it looks like this PR #30376 changed a lot in BackfillJobRunner, Maybe there is something we missed in there. Additionally, these changes are not even mentioned in the RELEASE_NOTES.rst.

reviewable stages: #30255 -> #30302 -> #30308 -> this PR.

How to reproduce

1. any simple dag file
2. run airflow dags backfill with above dag's id
3. check webserver's Browse > Jobs, and limit the Job Type to BackfillJob. or just check record in job table when JobType = 'BackfillJob'

This issue can be reproduced in versions from 2.6.0 to the latest version.

Operating System

MacOS 14.4.1

Versions of Apache Airflow Providers

None

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[pateash]

I can take a look at this, If no one else is picking up.

[Jie211]

Hi @pateash
Thanks for offering to help with this issue!
I've been quite busy and haven't managed to revisit the code.
Any assistance you can provide would be greatly appreciated.
Let me know if you need any details to get started.

[pateash]

Thanks @Jie211,
Please feel to assign this to me,
I will take a look at this

[potiuk]

assigned

[Benazeer45]

While reproducing, I noticed that when running airflow dags backfill, no records appear under BackfillJob in the Jobs table. I checked the available jobs, and BackfillJob is missing. Is there any known issue or workaround for this?

[nailo2c]

It looks like Airflow 3 now stores backfill details in the backfill and backfill_dag_run tables.

@potiuk, should we still record BackfillJob entries in the job table?