Support multiple APIs and version numbers per package

[LilithHafner]

I want to depend on a package's internals and still safely benefit from automatic update propagation using SymVer. To do this, I want the ability to be able to define multiple APIs for a single package.

Let's consider a case study. DataStructures.jl defines DataStructures.MultiDict, and many public functions to work with it including modification and queries. Now let's say I'm making a package and have an uncommon use case that warrants using DataStructures.MultiDicts but also requires a specialized high-performance implementation of an obscure operation that DataStructures.jl doesn't support. I read the implementation of DataStructures.MultiDict and write some self contained code that takes a DataStructures.MultiDict, and efficiently performs my operation by directly manipulating its internal fields. Now I have several choices: 1) add my code to the public API of DataStructures.jl 2) add the internal fields of DataStructures.MultiDict to the public API of DataStructures.jl 3) use the internals and declare a dependency on the exact version of DataStructures "=0.18.13" requiring me to manually green-light all updates to DataStructures.jl before users of my package can use them 4) use the internals and, declare permissive compat bounds such as "^0.18.13", and hope for the best even though it would be totally valid for DataStructures.jl to change their internals and break my code in 0.18.14.

There are cases when none of these are great options. I'd like to add yet another option: declare a dependency on DataStructures.DictInternals.

In implementation, packages may declare additional version numbers in their Project.toml, like so

name = "DataStructures"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.18.13"
DictInternals.version = "3.0.9"

And define the scope of "DictInternals" in their documentation. In this example, let's say the API of DictInternals includes DataStructures.MultiDict, including all its fields and inner constructors, but does not include any other functions.

Other packages can depend on these alternate version numbers in addition to or instead of the primary version number with

[deps]
DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"

[compat]
DataStructures = "0.18.13"
DataStructures.DictInternals = "2.0.6, 3"

Now, in writing my package, I declare a dependency on DataStructures with version "0.18.13" and also declare a dependency on DataStrucutres.DictInternals with a version "2.0.6, 3" (carrot specifiers are implicit).

Now, if DataStructures.jl makes a non-breaking change that also doesn't change dictionary internals, they can release it as

version = "0.18.14"
DictInternals.version = "3.0.10"

If they make some changes to dictionary internals that were not covered by their well-defined DictInternals API, then they can also release

version = "0.18.14"
DictInternals.version = "3.0.10"

But if they remove a field or add, remove, or change any invariants covered by DictInternals (such as the interpretation of a UInt8 metadata field), then that is a breaking change in DictInternals. For example, if they change from storing keys and values in separate arrays to storing them inline together, that would be breaking to DictInternals, even if it doesn't affect DataStructure.jl's public API. In this case, they would release:

version = "0.18.14"
DictInternals.version = "4.0.0"

In the first two cases, Pkg would automatically mark my package as compatible with the new version of DataStructures.jl because there were no breaking changes. In the last case, Pkg would mark my package as incompatible but still mark everyone who doesn't depend on DictInternals's packages as compatible.

[dalum]

I quite like this idea. I think a more common use-case for a feature like this, would be to support relying on a subset of the public API that doesn't change during a breaking version bump, though. For example, if you only need DataStructures.MultiDict, you may declare dependencies like:

[compat]
DataStructures = ">= 0.18.13"
DataStructures.MultiDict = "3"

allowing you to freely upgrade DataStructures, as long as MultiDict is left unchanged. In the bikeshed department, I would probably prefer a variation of something like:

[compat]
DataStructures = { version = ">= 0.18.13", MultiDict = "3" }

to make sure these are grouped together in the compat section. I also don't know if it's technically possible to register a package with a dot in the name, in which case DataStructures.MultiDict could be ambiguous?

[LilithHafner]

A couple notes,

Everything that is possible with this is also possible to do by adding wrapper packages that re-export subsets of the wrapped package's public and/or private API and which specify an = compat with the wrapped package and get new releases each time the wrapped package releases a new version. That's a lot of plumbing, though, and to reduce undeclared dependence on internals, I think it is worth making this as easy as possible.

relying on a subset of the public API that doesn't change during a breaking version bump

That's an interesting use-case, thanks for bringing it up! I'd denote it DataStructures.MultiDict = "3" or DataStructures = { MultiDict = "3" }, though. I don't think the >= compat specifier is necessary. Either you depend on something in DataStructures which is not covered by MultiDict so >= is too permissive and ^ is appropriate, or you don't depend on anything in setdiff(DataStructures, DataStructures.MultiDict) and the ">= 0.18.13" is an unnecessary restriction so long as DataStructures.MultiDict has been correctly versioned.

Sub-APIs may be added and removed in non-breaking releases of the main package's API without issue. The resolver would assume that a package with a dependency on DataStructures.DictInternals is incompatible with a version of DataStructures that does not declare a version number for DictInternals.

I also don't know if it's technically possible to register a package with a dot in the name, in which case DataStructures.MultiDict could be ambiguous?

I don't think this is possible.