Extension Structure ADR proposal

[cescoffier]

First draft (following #47074)

Ping: @gsmet @geoand @dmlloyd

[geoand]

I will have a look tomorrow

[cescoffier]

Yesterday, I drafted that proposal, and I gave some thought about it last evening (yes, I should have considered doing this in the opposite order).

I've two questions:

• JPMS name: Should only the runtime module have a JPMS name (and maybe spi), or should every module have one, as proposed? (@dmlloyd)
• In the case of an extension having no public API in its runtime module and having its public API in the spi module, what should be the root package of the spi module: io.<quarkus"quarkiverse>.<extension-name>.spi like proposed or just io.<quarkus"quarkiverse>.<extension-name>. The latter can introduce a split package risk if, at some point, the runtime module starts adding a public API.

[holly-cummins]

Related (on the linting side): quarkiverse/quarkiverse-devops#231

[dmlloyd]

• JPMS name: Should only the runtime module have a JPMS name (and maybe spi), or should every module have one, as proposed? (@dmlloyd)

IMO yes (every module should have a JPMS name) - we will have to tackle this eventually and it's better to have it done ahead of time. Perhaps our tooling can set this up automatically?

• In the case of an extension having no public API in its runtime module and having its public API in the spi module, what should be the root package of the spi module: io.<quarkus"quarkiverse>.<extension-name>.spi like proposed or just io.<quarkus"quarkiverse>.<extension-name>. The latter can introduce a split package risk if, at some point, the runtime module starts adding a public API.

I think spi, but I also think that the distinction between API vs SPI needs to be made crystal clear in the docs.

[Ladicek]

The spi module that contains runtime SPI seems backwards. Most extensions do not expose public API directly, they just expose the API of the underlying library/framework. But I know we do have extensions that have public API (Redis Client comes to mind).

In my personal opinion, public API of an extension should be either in the runtime module, under io.quarkus|quarkiverse.<extension>, or in a separate module (called api, not spi), in the same package.

The runtime part of the extension, which is not a public API, should always be in the runtime module under io.quarkus|quarkiverse.<extension>.runtime.

I see no reason to expose runtime SPI in a different manner than runtime API. They should both be in the same place. If the runtime API has a reason to delimit an SPI, it can do it in all the traditional manners -- we don't need an extra module in the extension just for a runtime SPI. If we need outside world to be able to depend on a runtime API of an extension without depending on the extension as a whole, the full runtime API of the extension should be in a separate module (again, called api, not spi).

My $0.017.

[cescoffier]

@Ladicek I think it's mostly a naming issue (what is not a naming issue these days 😄).

Let me try to clarify how we’re thinking about the separation between runtime, spi, and potential api modules.

API vs. SPI

It’s important to distinguish two concepts (as @dmlloyd said). Here is my attempt:

• API: Types (classes, interfaces) intended to be used by application code or other extensions to consume functionality. Think: RedisClient, annotations, utility methods, etc.
• SPI: Types that allow others to contribute to the extension’s behavior, typically via user implementations, configuration classes (customizer), or extension points. Think: custom resolvers, listeners, info contributor, etc.

We agree here: API is about usage, SPI is about participation.

The Coupling Dimension

(Not this is very badly explained in the draft as @geoand mentioned)

Where things get more subtle is around coupling: does the application (or consuming extension) require the extension to be present at runtime?

• Strongly coupled: If you use an API, like RedisClient, you clearly need the Redis extension present. This API can safely live in the runtime module under the root package.
• Loosely coupled: But if you’re contributing something optional (e.g., a type annotated for discovery, or consuming a CDI event), the extension might not need to be present. That’s where we want to avoid pulling in runtime transitively, why we have these SPIs into a spi module.

In other words, this separation is about allowing contributions without forcing runtime presence, more architectural than semantic (and that's my whole problem...).

Why not call it api?

You suggested that the public API should live in a module called api, not spi. It makes sense.

If we were trying to reserve api for consumption (i.e., the user-facing surface) and spi for contribution (i.e., extension points), it makes sense to clearly isolate both. The info extension does that very well.

• Should the spi module be named api (and contain both SPI and API in different packages)? In this case we will force the coupling.
• Should we have 2+1 separated modules (spi, api, runtime), it's a valid suggestion. Do we need that extra complexity ?

That's definitely open for debate and suggestions.

What about consumed CDI events?

That’s, for me, a great case where things blur: the application can consume events produced by an extension without needing that extension to be present. It feels like a runtime API, but from a dependency standpoint, it behaves more like an SPI (you can consume it, even if the source isn’t there).

In those cases, having the event types in the spi module makes sense, not because they are technically SPIs, but because we want to avoid runtime coupling.

What about?

• Public API used at runtime → runtime module, root package
• Extension points and loosely-coupled contribution interfaces → spi module, .spi subpackage
• Split based both on intent (API vs SPI) and coupling (runtime presence vs isolation)

WDYT?

P.S.: Realizing that this comment is almost as long as the initial ADR...

[geoand]

API is about usage, SPI is about participation.

+1

In other words, this separation is about allowing contributions without forcing runtime presence, more architectural than semantic (and that's my whole problem...).

+1

[Ladicek]

So in my eyes, SPI is part of API. API is what users are supposed to use. Some usages are consumption, some are participation, but it's all API. If you feel like there's a clear line between consumption and participation, I'll respectfully disagree; outside the most simple cases, participation often requires consumption. In other words, an SPI typically looks like

interface Participant {
    void doSomething(Context ctx);
}

where Context either is part of API directly, or provides access to parts of API indirectly. An attempt to separate the two into different modules is an exercise is futility.

[cescoffier]

@Ladicek We have an example of clean delimitation (like the info extension). However, let's imagine we can't have a clean cut (and the CDI event is an example of this ambiguity). Should we have an api module that contains both the API and SPI, but without binding the consumer to the extension? The consumer can decide to either force the extension to be present (by depending on the runtime module) or keep the coupling loose (by depending only on the API module).

[Ladicek]

However, let's imagine we can't have a clean cut (and the CDI event is an example of this ambiguity). Should we have an api module that contains both the API and SPI, but without binding the consumer to the extension?

Yes indeed, that's what I'm advocating for.

[aloubyansky]

It seems like a split of api(spi)/runtime could be justified if there could be multiple implementations of that API/SPI, i.e. different extensions implementing them. Otherwise, bundling it all in a single runtime module shouldn't be a big deal, should it?

[cescoffier]

Otherwise, bundling it all in a single runtime module shouldn't be a big deal, should it?

If we add the classes into the runtime module, we end up having a tight coupling. If we add them into an api module, yes, it should not be a big deal. The issue is not the split, it's the coupling.

[aloubyansky]

I may have missed it, what's the problem with coupling in cases when there is only one impl of a given API?

[cescoffier]

@aloubyansky it's mostly what I tried to explained here:

---

The Coupling Dimension

(Not this is very badly explained in the draft as @geoand mentioned)

Where things get more subtle is around coupling: does the application (or consuming extension) require the extension to be present at runtime?

• Strongly coupled: If you use an API, like RedisClient, you clearly need the Redis extension present. This API can safely live in the runtime module under the root package.
• Loosely coupled: But if you’re contributing something optional (e.g., a type annotated for discovery, or consuming a CDI event), the extension might not need to be present. That’s where we want to avoid pulling in runtime transitively, why we have these SPIs into a spi module.

In other words, this separation is about allowing contributions without forcing runtime presence, more architectural than semantic (and that's my whole problem...).

---

If we put API+SPI in the runtime module, consumers will be tightly coupled: the extension will be pulled.
If we put API+SPI in an api module, consumers can decide their coupling:

• depending on the api module will not pull the extension
• depending on the runtime module will pull the extension (the runtime module depends on the api module)

[aloubyansky]

Thanks. So strong coupling is still an extension developer choice.

[dmlloyd]

Having the API module be io.<quarkus|quarkiverse>.<extension-name>.api but having a package name of io.<quarkus|quarkiverse>.<extension-name> is wrong and will cause problems. Even more confusing when the runtime module is io.<quarkus|quarkiverse>.<extension-name> but in package io.<quarkus|quarkiverse>.<extension-name>.runtime. The package name and module name must match except for only very special situations.

What if we call the API JPMS module io.<quarkus|quarkiverse>.<extension-name>.api (if any) and the runtime JPMS module io.<quarkus|quarkiverse>.<extension-name>.runtime, with package names that match?

[cescoffier]

The fact that most extension public API are under io.<quarkus|quarkiverse>.<extension-name> is quite annoying.

In fact, if the extension has a public API in a runtime-api module, I would enforce that the runtime module does not contain io.<quarkus|quarkiverse>.<extension-name>, so basically, we should never have the case where both io.<quarkus|quarkiverse>.<extension-name> and io.<quarkus|quarkiverse>.<extension-name>.api exist in different modules.

[dmlloyd]

I think it's OK to have extensions which do not match the convention for package names, however the convention/spec should specify a scheme that is correct, else nothing will be correct. So we should recommend that the module should be ...api and the package should be ...api (or that the module and package for the API reside in a completely different, non-Quarkus package/module namespace, as long as the packages match the module name).

[gsmet]

I promised to have a look, sorry for the delay, was deep in other subjects (and PTOs!).

[gsmet]

For me, nothing in the deployment should be API. It's SPI.

[cescoffier]

If we consider the following definitions:

• API: Types (classes, interfaces) intended to be used by application code or other extensions to consume functionality. Think: RedisClient, annotations, utility methods, etc.
• SPI: Types that allow others to contribute to the extension’s behavior, typically via user implementations, configuration classes (customizer), or extension points. Think: custom resolvers, listeners, info contributor, etc.

and if we look at some of the deployment-spi, they are not spi. Typically, build items used as "phase barrier", or build item that allows you retrieving a build object. Maybe these cases are abuse of the system, but we have tons of things like this.

If we want to say: it's SPI only, it should only contain build items produced by a depending extension and consumed by the host extension.

[dmlloyd]

That's a good breakdown of the possible usages. But I think that things (build items?) which allow other extensions to consume functionality of an extension (as opposed to contribute to it) do fit better under the SPI heading rather than API, so @gsmet might be right here. There is little (nothing?) in deployment-[as]pi that applications can actually consume directly, so there'd be nothing left under "API" in this case.

[gsmet]

What @dmlloyd said is exactly what I meant.

Keeping spi for deployment has an additional benefit, in addition to being the right semantic IMO, we won't break all the extension ecosystem.

[mkouba]

There is little (nothing?) in deployment-[as]pi that applications can actually consume directly,...

Not applications but other extensions can - a config mapping with ConfigPhase#BUILD_TIME is IMO a good example of the deployment API...

[dmlloyd]

IMO config should not be public between modules, usually. I'd like to examine some cases where we are doing this to determine whether I'm wrong or whether we should be solving these problems some other way.

[cescoffier]

@aloubyansky @gsmet @geoand Could you have another look?

[aloubyansky]

I forgot, did we consider calling it runtime-dev? Theoretically, we may want to introduce build-dev too.

[cescoffier]

it makes sense to have it named runtime-dev as we have runtime-api and not just api.

[cescoffier]

We could revisit later if we need a build-dev, do we have an extension using that kind of module?

[gsmet]

I think we need to think about the transition path.

Because some changes might not be worth all the hassle they will generate (I'm thinking of the deployment -> build change for instance).

[gsmet]

I think we need to take a deep breath here and think about all the consequences of this change. It has relatively low value and could require a massive amount of work.

If we actually need relocations (which I'm not sure of), our BOM will be even more gigantic and it will slow down our Maven builds (it's already in the way as it is).

Also, we need to think about how we will support extensions that have not yet been migrated to the new model. This change impacts us but also impacts the whole ecosystem.

It's THE change that will trigger work on EVERY extension out there.

[gsmet]

Should we decide what to do here before finalizing the ADR?

[cescoffier]

I've reverted the renaming of the deployment and deployment-spi to build and build-spi.

[dmlloyd]

I think this is probably a mistake. All the concerns that were raised lately had already been addressed earlier. There's no urgency to force people to migrate their extensions. And capitulating to build tool conventions is a very slippery slope, since those can always change over time. Those were the main concerns IIRC and they were addressed.

[dmlloyd]

I think we need to think about the transition path.

Because some changes might not be worth all the hassle they will generate (I'm thinking of the deployment -> build change for instance).

But we don't need to do this urgently IMO. We're establishing a new convention, not a new technical mandate.

[Ladicek]

I don't have an iron in this fire, but:

1. I agree that we should keep deployment instead of renaming to build. Indeed build is a better name, but not significantly better. Also the Gradle argument makes perfect sense -- Gradle 1.0 was released more than a decade ago (and the first 0.x release goes ~5 more years into the past) and has been using the name build ever since. Just like Maven has been using target since its inception. That itself is a good enough reason to avoid naming an important directory build. (For the record, I think build is a fairly common name for a directory to which build outputs should be stored. IIRC, CMake also recommends using that name, although it doesn't enforce it.)

   If we actually really want to move away from deployment, I think builder would be a reasonable name. Anything that starts with build is gonna be somewhat clashing, plus "builder" immediately reminds me of the builder pattern, which is a completely different thing, but still.

2. I have no idea why we're trying to separate API and SPI so hard. Build items are not just produced, but also consumed by extensions. That itself make deployment-spi an API. If we want to make a naming change on deployment*, we should rename deployment-spi to deployment-api. Not insisting on it -- just like above, it is a better name, but not significantly better.

[andreaTP]

Can we have a definition of what an "extension" is? And possibly what's not?

[andreaTP]

I maintain a few "extensions" doing codegen, in most cases the pattern used is:

• codegen -> basically standalone (similar to a Maven plugin)
• runtime -> often used only to declare transitive dependencies
• deployment -> sometimes is just the default place that contains the codegen

I'm still unsure where each part of the code is supposed to go, do you have examples of repositories that already migrated(or open PRs) to this new structure?

[andreaTP]

Suggested change

	Extensions contributing to or using build items from other extensions should depend on `build-spi` modules.
	Extensions contributing to or using build items from other extensions should depend on `deployment-spi` modules.

[dmlloyd]

I don't have an iron in this fire, but:

1. I agree that we should keep deployment instead of renaming to build. Indeed build is a better name, but not significantly better. Also the Gradle argument makes perfect sense -- Gradle 1.0 was released more than a decade ago (and the first 0.x release goes ~5 more years into the past) and has been using the name build ever since. Just like Maven has been using target since its inception. That itself is a good enough reason to avoid naming an important directory build. (For the record, I think build is a fairly common name for a directory to which build outputs should be stored. IIRC, CMake also recommends using that name, although it doesn't enforce it.)

We aren't talking about naming a directory though, we're talking about naming a module. And ruling out module names because one build system claimed one of its components as a directory name - as if they were .com domains in 1995 - is definitely the tail wagging the dog.

Everyone agreed that in the absence of Gradle, "build" is a better name than "deployment" for the submodule. So, the module should be called "build". If someone wants to develop an extension using Gradle (which is, as we know, a minority case), they can name the subdirectory something else and the world will continue to turn.

Dismissing the review simply because the proposal is now materially different from what I approved.

[geoand]

I know people are not going to like me saying the following and I fully expert to be scolded for mentioning it, but here it goes...

Changing the directory from deployment to build is not going to be great for LLMs working with Quarkus code... All those models have seen tons of Quarkus code that all follows the same structure and now we are thinking about change it, making it even harder for the to reason about and produce good Quarkus code.
I completely realize this argument could be used for anything when ones argues to preserve the Status Quo, but I am worried that more and more people will rely on LLMs in the future and our architecture already makes it harder than our competitors to create advanced integrations without the extra confusion of a rename.

[dmlloyd]

I know people are not going to like me saying the following and I fully expert to be scolded for mentioning it, but here it goes...

Changing the directory from deployment to build is not going to be great for LLMs working with Quarkus code... All those models have seen tons of Quarkus code that all follows the same structure and now we are thinking about change it, making it even harder for the to reason about and produce good Quarkus code. I completely realize this argument could be used for anything when ones argues to preserve the Status Quo, but I am worried that more and more people will rely on LLMs in the future and our architecture already makes it harder than our competitors to create advanced integrations without the extra confusion of a rename.

Scolding incoming. :-) I have a hard time believing that this would be a significant factor one way or the other, honestly. And if it is a factor, I have a hard time believing that it will still be a factor a week, month, or year later. And I think that there's an equal chance that not changing it will be worse for AI.

I tend to believe that making AI "understand" our systems is the exact same kind of problem as making extremely naive junior developers understand our systems. Put another way, if we want cogent AI results, I think the number one thing we can do is make sure our docs are cogent, and I believe that means extremely unambiguous grammar and sentence structure, very clear ideas and principles, etc. The second we start making a single coding decision based off of what we think the impact on AI will be, we've lost in such a profound way that I can't even begin to describe it.

Principles first!

[geoand]

Yeah, I can't say I disagree. It's just something that in this day and, we'll likely have to be aware of more and more

[cescoffier]

Let me clarify my last changes. The idea was to split the ADR proposal into two parts:

• This ADR is as it is right now.
• Another ADR studying the possibility of switching from deployment to build-xyz and potentially doing that for Quarkus 4.

The goals of this second ADR/exploration are:

• finding the right name (build is going to be a no-go, build-time maybe)
• looking at the consequences (BOM explosion, readability, understandability)
• potential migration path

[FroMage]

I arrive late in this review, but I've searched both the other github discussion and this PR's discussion page, and have not found the rationale for splitting codegen into its own dependency?

Is it just a diehard tidying up of separating things, or is there a modularity reason?

Most of my extensions have codegen, so I'm having a hard time thinking of extensions without codegen, but also extensions with only codegen, or a reason why codegen could have different dependencies, or why someone might want to depend on a codegen module.

To me, so far, codegen are very tightly bound to build steps. I do agree that they could be in a standard package separate from build steps, but don't see the point of making an extra module.

Build items, yes, I do agree that they should be split into their own package, and it's good to make a distinction between internal build items and SPI build items. I'm not sure they require a separate module, but I guess it's useful for decoupling and dependencies?

Anyway, I'm wondering if the codegen split is purely driven by an obsessive requirement to sort things in separate boxes, or if there's an underlying technical reason?

[cescoffier]

Codegen is for things like Avro/Protobuf. These are command-line tools you need to run before the build to produce source code (.java, not bytecode) that the application code will use. We cannot do that at build time, because you won't reach "build time" (compilation will fail). We cannot use an annotation processor, because it's not annotations or even Java-based.

[dmlloyd]

Compilation of .java files should be a part of our workflow. The only reason it isn't is concessions to Maven/Gradle IMO...

[andreaTP]

We cannot use an annotation processor, because it's not annotations or even Java-based.

This is not technically correct, I quickly got a POC together to show that is completely possible to generate code out of an annotation processor without the need of annotating anything.

https://github.com/andreaTP/poc-source-codegen-annot

I stand with @FroMage and believe that removing artificial distinctions would be a plus if that's still open for discussion.

[aloubyansky]

+1 to compilation included in our workflow.
In this case, it's not only about compilation, it's about generating sources by calling other tools. Which arguably should also be a part of our workflow too, today it's implemented in a very different way though.

[FroMage]

It's entirely my fault I guess:

Contains code generation utilities such as code pre-processors invoked at build time

This is clear.

[andreaTP]

We tried to include these in our workflow, the developer experience was plain horrible: you had to delete classes from your code to enable progress.

What's the issue?
Seems like a technical implementation limitation rather than anything else ...

[andreaTP]

Anyway, looks like that ship has already sailed, no further comments.

[geoand]

The scope of the ADR is already pretty large, and the ADR has already changed significantly since its inception, so IMO we should move discussions about including compilation into a separate discussion

[dmlloyd]

The tangent is my fault, sorry. To bring it back around to Stef's original confusion then: perhaps we should call it sourcegen. Then if/when we ever roll it into the workflow we can deprecate and remove the concept.

[FroMage]

I am left a little skeptical about package rules for private vs public:

• io.quarkus.foo.runtime.internal|impl (private, historical)
• io.quarkus.foo (public, historical)
• io.quarkus.foo.api (public, new)
• io.quarkus.foo.deployment (private, historical)
• io.quarkus.foo.deployment.spi (public, new)

My problem is that sometimes it's a higher package that is public, sometimes it's a deeper package. This does not seem consistent or intuitive.

[cescoffier]

First, we recommend using different modules for public and private packages.

Then, io.quarkus.foo.runtime is more of something we inherit from, but, hopefully, would be replaced by .api or .impl (currently, we have a mix of both).

For deployment, I'm not against adding .impl or .internals.

[FroMage]

[Note that I've updated my comment's list because it was wrong]

Oh, so it's …<extention-name>.runtime.internal or …<extention-name>.impl? I read it as …<extention-name>.runtime.internal or …<extention-name>.runtime.impl 🤔

[dmlloyd]

Let me clarify my last changes. The idea was to split the ADR proposal into two parts:

• This ADR is as it is right now.
• Another ADR studying the possibility of switching from deployment to build-xyz and potentially doing that for Quarkus 4.

OK. I don't really understand why we would wait though. Changing the policy doesn't mean we must change everything immediately.

The goals of this second ADR/exploration are:

• finding the right name (build is going to be a no-go, build-time maybe)

OK, this is still wrong AFAICT. If build is the best name then build shall be the name. If Gradle conflicts, we rename the directory to something else when using Gradle. That's it; I don't understand why this would be even a little controversial.

• looking at the consequences (BOM explosion, readability, understandability)
• potential migration path

[gastaldi]

The fact that Gradle chose build compared to what other build tools use is unfortunate, but I guess we'd have the same problem if we consider different build tools, so I believe what @dmlloyd says makes sense

[mkouba]

Shouldn't this be something like io.<quarkus|quarkiverse>.<extension-name>.runtime? Because otherwise io.<quarkus|quarkiverse>.<extension-name>.api etc. are sub-packages (ok, I know that nothing like this is formalized in Java) of this root and it might confuse people and tools.