OCI artifact manifest, Phase 1-Reference Types #29
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,4 +2,4 @@ | |
| + Derek McGowan <[email protected]> (@dmcgowan) | ||
| + Mike Brown <[email protected]> (@mikebrow) | ||
| + Stephen Day <[email protected]> (@stevvooe) | ||
| + Joey Schorr <jschorr@redhat.com> (@josephschorr) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,66 +1,17 @@ | ||
| # OCI Artifacts (Experimental) with OCI Artifact Reference Support | ||
|
|
||
| This is an experimental branch of oci artifacts, validating the [oci.artifact.manifest spec][oci-artifact-manifest-spec] support of reference types. Reference types are required to meet the [Notary v2 Requirements][nv2-requirements] for not changing the target digest or tag, and the [Notary v2 Scenarios][nv2-scenarios] for content movement within and across registry implementations. Reference types enable a wider range of scenarios, including secure supply chain artifacts that may be represented as a graph as they move across environments. | ||
|
|
||
|  | ||
|
|
||
| ## Table of Contents: | ||
|
|
||
| - [OCI Artifact Manifest Overview][oci-artifact-manifest] | ||
| - [OCI Artifact Reference Type Manifest Spec](./artifact-reftype-spec.md) | ||
| - [ORAS experimental support for oci.artifact.manifest references][oras-artifacts] to `push`, `discover`, `pull` referenced artifact types. | ||
|
|
||
| [oci-artifact-manifest]: ./artifact-manifest.md | ||
| [oci-artifact-manifest-spec]: ./artifact-reftype-spec.md | ||
| [nv2-requirements]: https://github.com/notaryproject/notaryproject/blob/main/requirements.md | ||
| [nv2-scenarios]: https://github.com/notaryproject/notaryproject/blob/main/scenarios.md | ||
| [oras-artifacts]: https://github.com/deislabs/oras/blob/prototype-2/docs/artifact-manifest.md | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # OCI Artifact Manifest Spec (Phase-1 Reference Types) | ||
|
There was a problem hiding this comment. This seems like it's been hashed out in the past, but what's to stop us from using the existing image manifest spec and adding only the additional fields needed to enable references? In my mind, this would just be I think that we should at least list out the reasons why we think we need a new manifest type and why we can't use the existing spec. Especially with the conversations going on about another "object manifest" type in the future, I'd really like to keep the number of schemas that registries have to implement to a minimum. Adding fields to the existing spec seems like it might be simpler and reduce the overall churn over the next few years. There was a problem hiding this comment. One major issue is that if any object can have a reference (which is a reverse reference for gc) and also a normal link to a manifest, we can no longer guarantee that there are no cycles in the garbage collection graph, and gc becomes intractable. There was a problem hiding this comment. My understanding was that the garbage collector will not be using the There was a problem hiding this comment.
I'm not sure if I 100% agree with this, but I do agree that it becomes more complicated. It's hard to really discuss because there are so many different GC policies. I don't like that this doc prescribes GC behavior, though, as it limits the power of how this kind of relationship could be used. There was a problem hiding this comment. The limitation is simply a minimum scoped set of scenarios we're confident we can support until we have the time to work through all the scenarios in #37. There was a problem hiding this comment. @justincormack is there a restriction preventing an ArtifactManifest from referencing an ArtifactManifest? There was a problem hiding this comment. Nope. See the example where an SBOM is signed. The SBOM is an
|
||
| The OCI artifact manifest generalizes the use cases of [OCI image manifest][oci-image-manifest-spec] by removing constraints defined on the image-manifest such as a required `config` object and required & ordinal `layers`. It then adds a `subjectManifest` property supporting reference types. The addition of a new manifest does not change, nor impact the `image.manifest`. It provides a means to define a wide range of artifacts, including a chain of related artifacts enabling SBoMs, on-demand loading, signatures and metadata that can be related to an `image.manifest` or `image.index`. By defining a new manifest, registries and clients opt-into new capabilities, without breaking existing registry and client behavior or setting expectations for scenarios to function when the client and/or registry doesn't yet implement the new capabilities. | ||
|
|
||
| To enable a fall 2021 focus on supply chain security, **Phase 1** will narrowly focus on Reference Type support, giving time for further generalization with less time constraints. | ||
|
|
||
| For usage and scenarios, see [artifact-manifest.md](./artifact-manifest.md) | ||
|
|
||
| ## Example OCI Artifact Manifests | ||
|
|
||
| The following are Phase 1 examples: | ||
|
|
||
| - [`net-monitor:v1` oci container image](./artifact-manifest/net-monitor-oci-image.json) | ||
| - [`net-monitor:v1` notary v2 signature](./artifact-manifest/net-monitor-image-signature.json) | ||
| - [`net-monitor:v1` sample sbom](./artifact-manifest/net-monitor-image-sbom.json) | ||
| - [`net-monitor:v1` nydus image with on-demand loading](./artifact-manifest/net-monitor-image-nydus-ondemand-loading.json) | ||
|
|
||
| ## OCI Artifact Manifest Properties | ||
|
|
||
| For **Phase 1**, an artifact manifest provides an optional collection of blobs and a reference to the manifest of another artifact. | ||
|
|
||
| - **`schemaVersion`** *int* | ||
|
There was a problem hiding this comment. I would just drop this, as it is a vestige of old types. No need to carry this into new versions. There was a problem hiding this comment. can I get a hell yes! ( to removing it) There was a problem hiding this comment. I've always wondered how There was a problem hiding this comment. All of everything we see today is There was a problem hiding this comment. Just asking for clarity. There was a problem hiding this comment.
Implementations generally shouldn't be trying to peek inside an artifact in order to understand how to process it. By that point, it's often too late. Instead, content should generally be presented alongside a descriptor, which contains the mediaType of the content. For a registry, these descriptors are sometimes communicated via HTTP headers by the client on push and the server on pull. Sometimes, the descriptors are embedded within other content, e.g. when pulling a manifest list, you select one of the If you try to use the embedded mediaType to process everything, you'll probably end up having to parse content twice. Once to find its mediaType, then again to parse based on the mediaType. Very old clients didn't send or process HTTP headers sufficiently to behave properly, so this schemaVersion existed to differentiate between schema 1 and schema 2 images. Now that we live in a universe with a nice type system, that shouldn't be necessary. If you were to add a schemaVersion field to a new kind of artifact, you absolutely don't want to set it to There was a problem hiding this comment. Thanks Jon, |
||
|
|
||
| This REQUIRED property specifies the artifact manifest schema version. | ||
| For this version of the specification, this MUST be `3`. The value of this field WILL change as the manifest schema evolves. Minor version changes to the `oci.artifact.manifest` spec MUST be additive, while major version changes MAY be breaking. Artifact clients MUST implement version checking to allow for future, yet unknown changes. Artifact clients MUST ignore additive properties to minor versions. Artifact clients MAY support major changes, with no guarantee major changes MAY impose breaking changing behaviors. Artifact authors MAY support new and older schemaVersions to provide the best user experience. | ||
|
|
||
| - **`mediaType`** *string* | ||
|
There was a problem hiding this comment. Embedding the If we want to have this ability, we should come up with a different name. Using the mediaType to set the content of "THIS" document will be confusing, since we use it in both roles. There was a problem hiding this comment. I'm not sure how this correlates with the ^ q&a. There was a problem hiding this comment. This is a good thread to tug on, for some background: opencontainers/image-spec#411 (review) |
||
|
|
||
| This field contains the `mediaType` of this document, differentiating from [image-manifest][oci-image-manifest-spec] and [oci-image-index]. The mediaType for this manifest type MUST be `application/vnd.oci.artifact.manifest.v1+json`, where the version WILL change to reflect newer versions. Artifact authors SHOULD support multiple `mediaType` versions to provide the best user experience for their artifact type. | ||
|
|
||
| - **`artifactType`** *string* | ||
|
There was a problem hiding this comment. How is an artifactType different than a mediaType? Why not just have a mediaType? There was a problem hiding this comment. as you determined above this spec reads as it is looking at "this" format (manifest) to be defined by mediaType and artifactType is a sub-type within artifacts registered with iana.. with certain additional rules and content types expected in the blobs (artifact blobs) There was a problem hiding this comment. I'm may be missing some level of detail to the question here. This decouples the registry from knowing about the specific artifacts, just like a filesystem knows how to store files, but doesn't care about the
There was a problem hiding this comment. I agree with Steve here, having a spec that allows any kind of typed object to be put in a field requires registry implementations to become stricter with what they accept compared to today. For example, registries might reject an otherwise well-formed and to spec OCI Image because they don't recognize the config blob's media type, since they need to verify each object is something they know how to deal with in relatively fundamental ways. For distribution, this is as basic as knowing where this object should be located at all, given the digest. I don't believe ignoring the unknown mediatypes is appropriate for registries accepting pushes either, as this has implications for lifecycle management. For example, I believe some implementations accepted buildkit caches, but since they are OCI Image Index which presumably contained manifests, the associated blobs were deleted during garbage collection. This is pretty opaque to the end user, and I believe that registries which do lifecycle management of manifests and their referenced objects should return Having a broad category of "manifests" and "blobs", which are distinctions that I think most people here understand conventionally, allow registries to have a basic understanding of the level of expectations around an object without having to know the media type beforehand. There are already separate API routes for blobs and manifests, so this distinction exists within the v2 API spec today. For manifests, this does mean that registry implementations will have to know about the mediatype to read the manifest content correctly since registry implementations are expected to perform actions that require inspecting the manifest, such as validation. However, for an object that's classified as a blob, the registry implementation is safe to make certain assumptions such as, that that object will not reference other objects in a way that's relevant to the registry, and that it should not be parsed as JSON. Having this clarity would allow registries and clients enough wiggle room to work together without having to keep in lock step. There was a problem hiding this comment. @deleteriousEffect I generally agree with you, but I don't see how this relates to |
||
|
|
||
| Phase 1 of the OCI Artifact spec will support reference types to existing [OCI Artifacts][oci-artifacts]. The REQUIRED `artifactType` is unique value, as registered with iana.org. See [registering unique types.][registering-iana]. The `artifactType` is equivalent to OCI Artifacts that used the `manifest.config.mediaType` to differentiate the type of artifact. Artifact authors that implement `oci.artifact.manifest` use `artifactType` to differentiate the type of artifact. example:(`example.sbom` from `cncf.notary`). | ||
|
|
||
| - **`blobs`** *array of objects* | ||
|
There was a problem hiding this comment. note to self.. need to officially define layer in the distribution spec.. missed it. There was a problem hiding this comment. not sure if I like defining blobs here gut says CAS should own blobs.. and image would do layers is array of blobs.. having ordinality .. over here in artifacts spec.. this would be artifacts or artifactBlobs is array of blobs where ordinality is optional.. There was a problem hiding this comment. course we could do the same thing by changing objects to CAS objects or something similar.. There was a problem hiding this comment. I would prefer something pointing towards their use. Should this be "subjects"? How does it relate to the manifest? There was a problem hiding this comment. We seem to be discussing the differences between a manifest, which represents an artifact (container image, helm, wasm, opa, ...) and the content that makes up that artifact, represented as These are distinct types. They both use descriptors to define how they're persisted as CAS objects, but they do have different meanings, where the registry, and a client processes a manifest for it's content, but should just store and serve |
||
|
|
||
| An OPTIONAL collection of 0 or more blobs. The blobs array is analogous to [oci.image.manifest layers][oci-image-manifest-spec-layers], however unlike [image-manifest][oci-image-manifest-spec], the ordering of blobs is specific to the artifact type. Some artifacts may choose an overlay of files, while other artifact types may store indepdent collections of files. | ||
|
|
||
| - Each item in the array MUST be a [descriptor][descriptor], and MUST NOT refer to another `manifest` providing dependency closure. | ||
| - The max number of blobs is not defined, but MAY be limited by [distribution-spec][oci-distribution-spec] implementations. | ||
| - An encountered `descriptor.mediaType` that is unknown to the implementation MUST be ignored. | ||
|
|
||
| - **`subjectManifest`** *descriptor* | ||
|
There was a problem hiding this comment. I would just call this There was a problem hiding this comment. How would that be different than another entry in the |
||
|
|
||
| An OPTIONAL reference to any existing manifest within the repository. When specified, the artifact is said to be dependent upon the referenced `subjectManifest`. | ||
| - The item MUST be a [descriptor][descriptor] representing a manifest. Descriptors to blobs are not supported. The registry MUST return a `400` response code when `subjectManifest` is not found in the same repository, and not a manifest. | ||
|
|
||
| - **`annotations`** *string-string map* | ||
|
|
||
| This OPTIONAL property contains arbitrary metadata for the image manifest. | ||
| This OPTIONAL property MUST use the [annotation rules](annotations.md#rules). | ||
|
|
||
| See [Pre-Defined Annotation Keys][annotations] | ||
|
|
||
| ## Push Validation | ||
|
|
||
| Following the [distribution-spec push api](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#push), all `blobs` *and* the `subjectManifest` descriptors SHOULD exist when pushed to a distribution instance. | ||
|
|
||
| ## Lifecycle Management | ||
|
|
||
| For Phase 1, artifact types will be limited to reference types. A reference type is an artifact that doesn't have a lifecycle unto itself. A container image is said to have an independent lifecycle. A reference type, such as an SBoM or signature have a lifecycle tied to the `subjectManifest`. When the `subjectManifest` is deleted or marked for garbage collection, the defined artifact is subject to deletion as well. A distribution instance SHOULD delete, (refCount -1) the artifact when the `subjectManifest` is deleted. | ||
|
|
||
| ### Tagged `referenceTypes` | ||
|
|
||
| As signatures and SBoMs are not considered independent artifact types, they SHOULD NOT have a tag, simplifying the lifecycle management. As the `subjectManifest` is marked for deletion (refCount=0), the `referenctType` is also marked for deletion (refCount -1). However, these artifacts MAY have tags as future versions of the artifact manifest MAY support independent types. | ||
|
There was a problem hiding this comment.
@nishakm see here and above There was a problem hiding this comment. I think we all recognize adding any content added to registries needs enough info to manage its lifecycle. This includes GC on incomplete uploads or enough information for users to delete their content. While registries implement GC and deletion management differently, we're just making sure we've provided the right level of information in the manifests to enable lifecycle management and set user expectations for consistency across registries. For phase 1, we're scoping this tightly so we can see how this evolves over time. For instance, we definitely want to support independent artifacts with |
||
|
|
||
| [oci-artifacts]: https://github.com/opencontainers/artifacts | ||
| [oci-config]: https://github.com/opencontainers/image-spec/blob/master/config.md | ||
| [oci-image-manifest-spec]: https://github.com/opencontainers/image-spec/blob/master/manifest.md | ||
| [oci-image-manifest-spec-layers]: https://github.com/opencontainers/image-spec/blob/master/manifest.md#image-manifest-property-descriptions | ||
| [oci-image-index]: https://github.com/opencontainers/image-spec/blob/master/image-index.md | ||
| [oci-distribution-spec]: https://github.com/opencontainers/distribution-spec | ||
| [media-type]: https://github.com/opencontainers/image-spec/blob/master/media-types.md | ||
| [artifact-type]: https://github.com/opencontainers/artifacts/blob/master/artifact-authors.md#defining-a-unique-artifact-type | ||
| [registering-iana]: ./artifact-authors.md#registering-unique-types-with-iana | ||
| [descriptor]: https://github.com/opencontainers/image-spec/blob/master/descriptor.md | ||
| [annotations]: https://github.com/opencontainers/image-spec/blob/master/annotations.md | ||
Uh oh!
There was an error while loading. Please reload this page.