Put AIP-52 setup/teardown tasks behind feature flag

[jedcunningham]

We aren't going to land AIP-52 in time for 2.6, so put the authoring api behind a feature flag. I've chosen to put it in airflow.settings so users can set it in airflow_local_settings, or set it via env var.

[potiuk]

We should do the same with AIP-44 I guess.

What's the idea about running tests - should AIRFLOW_ENABLE_AIP_52 be still enabled in CI ? I'd say yes as we are going to continue developing it but then we should disable it in the 2.6 branch so that the tests are not run there? Is my thinking right? Should we add it then in in ci.yml at the top of the workflow?

[jedcunningham]

I was thinking about the CI side of this this morning, and I could make an argument either way. I went for "off in main for now" as that'd match what non-AIP-52 contributors care about and would run locally, but I'm happy to enable it for main and disable it in the 2.6 branch though. Commit incoming.

[jedcunningham]

Actually, I'll make that change after I get a successful run here, just to make sure I didn't miss any tests relying on it.

[potiuk]

Actually, I'll make that change after I get a successful run here, just to make sure I didn't miss any tests relying on it.

Went the same route in my PR :) #30510 - will get it succeed and flip the flag then.

[potiuk]

I was thinking about the CI side of this this morning, and I could make an argument either way.

My argument is that if we don't enable it in main, then those contributors who don't care might (accidentally) break any work done by those who work on the in-progress AIP, so having it in main is a safeguard not to break what's already in main.

[jedcunningham]

Okay, this is ready for a final review now 👍

[potiuk]

FYI: with https://github.com/apache/airflow/pull/30510/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR965 we can both - have cake and eat it too @jedcunningham

[potiuk]

OK. I merged my change first - this one will need conflict resolution now on settings, but after rebase, it will also run a separate job with AIRFLOW_ENABLE_AIP_52: "false" on top of all the tests run with it set to "true"

[jedcunningham]

Nice, good call @potiuk!

[potiuk]

Nice, good call @potiuk!

You might want to rebase AGAIN to account for just merged #30521 that actually makes a difference for those two jobs (the variables were not propagated to docker environment so they had no effect).

[potiuk]

Nice. I see skipped AIP-52 tests in the separate job

[potiuk]

And they are not skipped in regular tests . All looks cool