Proposal: Add a manifest list API

[sudo-bmitch]

This is pulling out some comments made in #22 and a bit related to #114 . It would be useful to have some kind of /v2/<name>/manifests/list API similar to what we have for tags today. Returned from that should be all manifest digests within that repository.

This can be useful for building user scripts to implement the GC policy outside of the registry, looking for dangling manifests that do not have any tag pointing to them, and calling the manifest delete API when the user defined criteria is met. For example, a user script could examine the manifest and referenced config, looking for labels indicating the image was part of a nightly build, and remove any manifests pointing to a 2 week or older build.

[jonjohnsonjr]

From #114 (comment):

It would be useful to get a list of json including metadata

Emphatic yes, but I'd propose returning a list of descriptors instead of just mediaType and digest: #22 (comment)

Whether that's in the form of an image index or just a json array or even json-lines, I don't care that much, but I would really love for this to be available. cc @justincormack this is related to what I was saying in the OCI call a couple weeks ago

[sudo-bmitch]

@jonjohnsonjr seeing the recording of that meeting triggered this for me, went looking back at #22 and realized this point was rather buried. Getting this as a json list of descriptors makes a lot of sense to me. Definitely don't want the list of strings like we have with tag/list today.

[justincormack]

Yes, descriptors seems correct.

[jonjohnsonjr]

seeing the recording of that meeting triggered this for me, went looking back at #22 and realized this point was rather buried

Excellent! That's why I brought it up 👍

From my earlier comment:

Whether that's in the form of an image index or just a json array or even json-lines, I don't care that much, but I would really love for this to be available.

I'll campaign a little bit for my image-index-as-the-API idea.

Pros

One nice feature of making this format be an actual image index (and not a list of descriptors), would be the ability to reuse code that knows how to deal with an image index already. So if you wanted to e.g. mirror an entire repository from one place to another, I already have code that can deal with pulling and pushing an image index. I can just reuse that.

Continuing the example, if you want to keep your mirror up to date, you might need to poll tag values every so often to make sure they haven't change. If we could just ask a registry to show us everything in one request, we can just poll once per repo instead of once per tag. Assuming we cache everything aggressively, this shouldn't even be that expensive.

If we wanted, we could even expose the digest of this top-level image index so that clients can essentially ask "hey, has anything in this entire repo changed?" with a HEAD request, without even having to actually look at the entire structure. This could also allow clients to ask for the state of a repo from the past, if registries allow you to GET the list of manifests by digest (though, that might be too expensive to keep around).

Cons

If we do expose the digest of this structure, then re-computing the digest on every push/delete can be somewhat expensive for registries, so they may not want to do that. I think that's fine -- perhaps the Docker-Content-Digest header for listing manifests is optional, and we can direct clients to explicitly look for etags as a cheaper means to do this.

Another downside is that pagination could be a little weird. For enormous repositories, we probably don't want to send the entire list in one response, and paginating the entries of an image index might frighten and confuse some clients (of course, this is something we can just specify, since it's an entirely new API).

Pagination also makes the digest thing weird -- is this the digest of the whole repo? Or just this page? We may just want to punt on "digest of a repo" stuff for now.

[SteveLasker]

Having consistency across all registries for discovering content is a definite need. Removing the _catalog API cleared the way towards an implementation that could work across registries.

Might I suggest we capture the various requirements for the listing API? There are lots of great designs, but only a few that would meet the requirements we outline. For instance, when we were iterating on the Notary v2 discovery prototypes, we needed a way to discover all the artifacts that were dependent upon another artifact. eg: what signatures exist for a given digest.
We played with a few ideas, including what the schema would be. I think we considered image-index, but realize that any listing API needs paging. If you look at the helm index.yaml example, it was fine when there were only a few charts. But, fails when there's lots of charts. Imagine an image-index that has hundreds, or thousands of manifests? Having an API that supports paging 0, 1 to infinity means we never break. We'd also, likely want to incorporate some sort of sorting (push-date, last-pull-date, ..., annotation value, ...). I could see different registries having different search criteria enabled, but the API should be consistent. I'd actually hope we all supported some basic set of functionality, but I'm a purist with a reality check.

Here's a hacked up Notary v2 prototype-1 version, pre the newly proposed oci.artifact.manifest. This was used to get signatures for an image. You could filter on mediaType, to only get cncf.notary.v2 artifact types. However, we didn't yet add a way to filter further on just the signatures for registry.acme-rockets.io

@aviral26 just made an update for the OCI Artifact manifest, but this is a prototype, based on the Notary v2 and Artifacts requirements.

• Code is here
• Usage is here

We need to iterative further for the notary & artifact scenarios, but I'd love to see this evolve OCI Artifacts and a View of the Future So, just suggesting we capture and iterate on a list of requirements for the listing API. Then, we're debating which design meets the requirements, vs. which "design is better".

Also note, Docker has transitioned docker/distribution to CNCF. It's now located at: distribution/distribution and we have a new set of active maintainers from GitLab, GitHub, Digitial Ocean, VMWare, Docker, that want/need to keep the innovations moving forward.

[jonjohnsonjr]

We'd also, likely want to incorporate some sort of sorting (push-date, last-pull-date, ..., annotation value, ...)

Filtering based on annotation values is interesting. I don't love the other examples because they impose additional storage requirements on registries. I worry that waiting for a perfect solution (i.e. requirements for features that don't exist) will block a massive improvement ~indefinitely.

Can we agree on a simple set of requirements with possible extensions?

I think at a minimum we need:

1. Return a list of descriptors
2. Allow that list to be paginated

It would be nice to have:

3. Tag info, either:
   1. Map from tags to digests (we could add this to /tags/list, maybe?)
   2. List of tags on these descriptors -- would need to add a field?

We should leave open for extension:

4. Sorting
   1. Decide on some syntax to allow for sorting by arbitrary things
   2. Decide on some standard set of suggested things by which registries could sort by, similar to pre-defined annotation keys.
5. Filtering
   1. Decide on some syntax for filtering by arbitrary things
   2. Decide on some standard set of suggested things by which registries could filter.

I really don't want to block on 4 and 5. To borrow your "cloud filesystem" metaphor, I agree that having a SQL-like API for querying the filesystem would be really nice, but as is our filesystem can't even list files or directories... this is bad.

We should keep filtering/sorting in mind so that implementations or specs don't preclude it, but I don't think it's mandatory, and I don't expect all registries to implement this stuff. It should be possible to implement a registry with zero logic, just static files. We should be able to define some optional querystring parameters this.

Similarly, I don't want to block this on notary concepts that are still prototypes.

[SteveLasker]

a SQL-like API

Oh no, you brought up winfs - yikes 😳

I like the extension model or the reserved for future space approach.

Having the full list in mind, allows us to design a multi-phased approach. It's like building the house, knowing you want to add a deck or garage later. If we can get a full list, we can prioritize, while reserving space in the design for known additions.

Might I suggest a PR for manifest-list-requirements.md that also articulates a bit of the scenarios? The PR can be in a manifest-list branch that we iterate upon until we have an actual spec proposal that we could then review and decide how we incorporate the requirements list back into the spec as needed.

[jonjohnsonjr]

Might I suggest a PR for manifest-list-requirements.md that also articulates a bit of the scenarios?

I'm happy to send a PR, but I'd like some feedback from other registry operators. Ideally, we could talk about this on the dev call. I would personally commit to implementing this for gcr.io and pkg.dev if I could get some kind of consensus and commitment from other registries. If this is missing something that anyone feels is a hard requirement, I'd like to know. If this is too onerous to implement for any registries, I'd like to know.

It would be good enough for me if I could get any two of {Docker Hub, ECR, ACR, Quay} to implement this, but otherwise it's just an additional registry-specific thing that adds no value to end users.

I don't want to shove every possible feature you might want into these APIs. These are meant to be the lowest common denominator, bare minimum things that are inoffensive to implement and maintain, i.e. undifferentiated heavy lifting. If your registry supports some awesome feature that nobody else has, I think it belongs in a proprietary API.

As is, the only way to list repositories is to use _catalog. We told people not to use that, but offer zero alternative, so they use it. For untagged images, it's even worse. There is undefined behavior around what happens -- do images just disappear? Do they stick around forever? How do you know what exists? We really need a standard way to expose this information.

I already proposed these two additional APIs in #22, but I still think they're good places to start. They are simple and aligned with existing APIs, so they should be familiar to users. They expose information that already exists while being open to extensibility.

Reader: I implore you not to bikeshed this. If things can be simplified to make this more likely to be implemented, I'd love to hear it. If you want to attach a use case for a hypothetical future thing that may never exist, please weigh the expected value of that use case against the decreased likelihood of us reaching consensus.

1. Descriptor listing API

Concretely, some structs:

// ManifestDescriptor describes the content of a given manifest object.
type ManifestDescriptor struct {
	// MediaType is the media type of the object this schema refers to.
	MediaType string `json:"mediaType,omitempty"`

	// Digest is the digest of the targeted content.
	Digest digest.Digest `json:"digest"`

	// Size specifies the size in bytes of the blob.
	Size int64 `json:"size"`

	// Annotations contains arbitrary metadata relating to the targeted content.
	Annotations map[string]string `json:"annotations,omitempty"`

        // Tags contains a list of tags associated with this object.
        Tags []string `json:"tags,omitempty"`
}

// ManifestDescriptorList is a list of manifest descriptors for a given repository.
type ManifestDescriptorList struct {
	// Manifests references manifest objects. 
	Manifests []ManifestDescriptor `json:"manifests"`
}

This is a trimmed down version of Index and Descriptor to omit anything that doesn't make snse, with Tags added.

It might make sense to keep specs.Versioned and Annotations in ManifestDescriptorList. It might also make sense to just use Index directly, but then we'd need to figure out how to do Tags. I've omitted Platform from the descriptor because it's not something that is known at push time.

As an example:

GET /v2/<name>/descriptors/list

{
   "manifests": [{
      "digest": "sha256:7a47ccc3bbe8a451b500d2b53104868b46d60ee8f5b35a24b41a86077c650210",       
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 2035,
      "tags": ["latest", "v1"],
      "annotations": {
        "org.opencontainers.image.created": "1985-04-12T23:20:50.52Z"
      }
   },{
      "digest": "sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e",       
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "size": 943
   },{
      "digest": "sha256:703218c0465075f4425e58fac086e09e1de5c340b12976ab9eb8ad26615c3715",       
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "size": 1201,
      "tags": ["v2"],
      "annotations": {
        "org.opencontainers.image.created": "2001-04-12T23:20:50.52Z"
      }
   }]
}

Note the second manifest in particular. It is untagged. This is completely unreachable today via the registry API. Also note that it doesn't have a org.opencontainers.image.created annotation. This is metadata that is sometimes available within an image's config file, but it doesn't make sense for a manifest list, really. If your registry cares enough to index an image's config file, it could expose this here.

If we wanted, we could define more optional annotations for things like push time, pull time, etc. Really anything. Vendor-specific annotations can use their own namespacing. This could also be a place for registries to surface user-specified metadata about artifacts. All of the annotations should be optional.

A registry may choose to populate a ton of annotations, but none should be required for conformance. Mandatory fields are digest, mediaType, size, and tags.

I've excluded blob descriptors from this list, as I don't think it makes much sense. Garbage Collection around blobs is pretty consistent across registries, and all blobs that are in the registry should be reachable through these manifests (ignoring GC strategies). If anyone thinks it makes sense to have a list of blobs accessible, I'd love to hear why, but I think it's out of scope.

Pagination should work identically to tags listing.

2. Repository listing API

This is the same as from _catalog:

type RepositoryList struct {
	Name string   `json:"name"`
	Repositories []string `json:"repositories"`
}

With the added "name" from TagList to make parsing things slightly easier for clients.

The only thing we need to do differently from catalog is make this work for a repository and not for a registry. Perhaps a top-level thing could exist, for certain registries, but it certainly shouldn't be required if it doesn't make sense (e.g. for GCR, where everything is namespaced as gcr.io/<your-project-id>/...).

As an example:

GET /v2/library/repositories/list

{
  "name": "library",
  "repositories": [
    "adminer",
    "aerospike",
    "alpine",
    "alt",
    "amazoncorretto",
    "amazonlinux",
    "arangodb",
    "backdrop",
    "bash",
    "bonita",
    "buildpack-deps",
    "busybox"
  ]
}

This should list repositories immediately under "library". For registries that support multiply-nested repositories, like GCR, you should be able to subsequently call GET /v2/library/bash/repositories/list to list any repositories under "library/bash". Non-nested registries could return 404 or an empty list.

Pagination should work identically to tags listing.

The only downside to this that I could see is that there's no place to stick arbitrary metadata like we would have in descriptors/list. Honestly, I think this is fine, but I'm open to feedback if someone disagrees. I don't mind having something like a RepositoryDescriptor, but it feels a bit silly. There's no digest for a repo, really (unless you go down the rabbit hole of treating a repo as an index), so this doesn't map cleanly onto existing concepts. Maybe it makes sense to have a top-level Annotations in a RepositoryList that just exposes the current repository's metadata, but I don't really think that's necessary. As is, this satisfies the requirement of being able to list all the contents of a registry, which is what I'm mostly aiming to do.

I don't think sorting or filtering are nearly as interesting for repositories as they are for descriptors.

[jonjohnsonjr]

Ah, look at the timing: https://www.docker.com/blog/open-sourcing-the-docker-hub-cli-tool/

I worry a bit that Docker will care less about implementing a standard thing for this.

[sudo-bmitch]

@jonjohnsonjr Big thumbs up from me. I would lean towards having a RepositoryDescriptor rather than a list of strings both for future proofing and as a place for vendor extensions to be added without breaking the spec. Types of metadata include things like the star count and pulls in Hub, which would allow them to support the spec in their hub-tool.

[jonjohnsonjr]

Types of metadata include things like the star count and pulls in Hub, which would allow them to support the spec in their hub-tool.

That seems reasonable to me, especially if it unblocks Docker Hub.

[SteveLasker]

I'll read through the detailed post above tomorrow. But I did want to quickly note the docker hub cli.
@justincormack and I talked about it a while back. Like all registries hoping for a standard, but needing to provide some tools for their customers, they had to release something to cope with the deletion of content. Basically, the issue behind the throttling and costs conversation.
When we talked, this exact point came up. How do we take the time to invest in a common API, so each registry doesn't need to invest their own, and customers have to deal with 6 different registry apis.

• If we can create a standard API set across all OCI compliant registries, my crystal ball shows a host of community tools worth investing in. 😄
• If we can't agree, the community will just look at us like idiots, and we'll have a set of random plug-ins that will never work as expected ☹️

[justincormack]

@jonjohnsonjr the reason the Hub cli tool is currently a standalone binary not built into Docker cli is the standardisation issue - I talked about this on the OCI call before Christmas. We want to release something that works across registries but its a mess now.

[jonjohnsonjr]

the reason the Hub cli tool is currently a standalone binary not built into Docker cli is the standardisation issue - I talked about this on the OCI call before Christmas.

Oh yeah, I think it's completely reasonable -- don't get me wrong. Before, Docker Hub was in an awkward position where there was no way to do this, so my hope was that y'all would be extra motivated to throw weight behind some standardization here. Now, y'all are just in the same boat as the rest of us :) there are good reasons to have a separate CLI, e.g. to expose Docker Hub specific stuff.

We want to release something that works across registries but its a mess now.

Absolutely agree, which is why I'm pushing on this.

There's this tragedy of the commons where all (most?) registries now have a bespoke way to do the same thing, which is often good enough for a customer, but it hurts the ecosystem. If I want to write a tool that lists stuff in a registry (say, spinnaker), my options are:

1. do really inefficient stuff with catalog and tag listing
2. pull in every registry's bespoke API or tool for doing this

That sucks for users, and I think OCI exists to solve this kind of problem.

cc @hdonnay @samuelkarp @bainsy88

Do y'all have any interest in fixing this?

[SteveLasker]

I don't love the other examples because they impose additional storage requirements on registries.

While true, we will need a change to support the requirements of Notary v2. We are focused on two dimensions:

• Minimize the changes required
• Maximize the opportunities with a change. If we're going to make a change, let's make it a generic change that supports Notary, Helm, CNAB, and other artifact types that have references.

minimum we need:
Return a list of descriptors
Allow that list to be paginated

This is a great start. I like the paginated list of descriptors, although the notary signature scenario likely calls for a slightly bit more info. I'll capture in the listing api requirements.

Tag info, either:
Map from tags to digests (we could add this to /tags/list, maybe?)
List of tags on these descriptors -- would need to add a field?

Also great. I think we all struggle for how to provide a history of digests to a given tag. As we move into the gated-mirror scenarios, I expect we'll see customers asking for "rollback" of a tag to a previous digest. When not if an update fails.

We should leave open for extension:

Having a list, which we can all prioritize will surely help. I suspect what might be a lower priority to some, might be a higher priority to others. So, hopefully, we can divide and conquer. Both, for the spec and the reference implementation.

our filesystem can't even list files or directories... this is bad.

Yup, hopefully, we finally have enough "pain" to invest in the gain.

I'd like some feedback from other registry operators.

With the newly formed CNCF distribution/distribution group, I've forwarded the links to these discussions. Hopefully, we'll get more engagement and feedback.

I suspect the Notary v2 work, that must land this year, will be enough of a compelling event to provide feedback, and be a compelling customer need that we'll make good progress. It's not just my hope to leverage Notary to achieve these list goals, rather it's a requirement to meet the Notary v2 requirements.

_catalog and untagged manifests...

Let's capture these in the requirements. I think all registries have proprietary APIs to handle these scenarios. I'm hopeful we can take our experience for what we like and don't like to make a spec'd api we can all implement.

I already proposed these two additional APIs in #22...

I'm going to do the PM thing and ask we start with a set of requirements. It's really helped defuse the debates over different designs. Rather than argue which design is better, we can argue which designs meet the prioritized requirements, with the right usability.

BTW, I do like the proposal to return a type ManifestDescriptor struct {. I'd just like to match it to a set of requirements. :)

@sudo-bmitch

a place for vendor extensions

This is a great point. We should also capture the requirements.

Types of metadata include things like the star count and pulls in Hub, which would allow them to support the spec in their hub-tool.

To balance the boil the ocean, I'm actually hopeful

I think OCI exists to solve this kind of problem.

Yuppppppp!!!!!

We can do this…