Working group for Reference Spec

[sudo-bmitch]

I'd like to propose a new working group to create a reference-spec that would define a reference in OCI. At present, this is just a convention, originated by Docker, where alpine gets mapped to docker.io/library/alpine:latest (where docker.io isn't even the real registry name). There is currently an implementation for this in distribution/distribution that many have used. I believe it would be useful for OCI to formalize this, and even include a Go implementation.

Other use cases I can think of (or have seen):

• OCI Layout (both to entries with just a tag, and those with a full registry/repo:tag name in the index.json)
• Immutable Tags
• Alternate layer formats like eStargz
• Querying for an artifact that refers another manifest (see wg-reference-types for more details)
• Querying content from a pull through cache serving multiple registries.

This PR will remain in draft state until there are a sufficient number of owners and stakeholders that volunteer to work on the project.

Signed-off-by: Brandon Mitchell [email protected]

[sudo-bmitch]

We discussed this during the OCI meeting at KubeCon today and decided it makes more sense to add this to image-spec instead of a separate repo. I'm closing this and we can coordinate with PR's to image-spec.
https://hackmd.io/nZzK_4AfRz-xgya6KkqseA

[sudo-bmitch]

Reopening to continue the discussion from the mailing list: https://groups.google.com/a/opencontainers.org/g/dev/c/PpzsQMOnda4

cc @vbatts @Jamstah @sajayantony

[sajayantony]

Thanks for reopening this @sudo-bmitch. I think clarifying meaning of reference in distribution and OCI layout is valuable since parsing that regex depending on context and language choice would make it easier if there was a clear definition in image spec (on which distribution indirectly depends on anyway). e.g. write up (but doesn't include the regex) - https://oras.land/docs/concepts/reference

[harche]

cc @lumjjb

[Jamstah]

There's not a lot in the minutes explaining the decision to include it in image-spec, can anyone remember the reasoning?

Personally, I think defining references separately makes sense. The image spec as written is very cleanly encapsulated around what an image is and not where an image is. It also gives more flexibility for the future, if we need to refer to things other than images. Putting it in the image spec would also tie up the lifecycles, and I'm not sure that's a good thing.

[sudo-bmitch]

There's not a lot in the minutes explaining the decision to include it in image-spec, can anyone remember the reasoning?

I believe @imjasonh weighed in on that. The impression I got was that creating a new spec is more effort than extending an existing one, and they felt that image spec was the closest fit. Given that I originally opened this to create a new spec myself, I tend to agree with you that it doesn't feel like a great fit.

Personally, I think defining references separately makes sense. The image spec as written is very cleanly encapsulated around what an image is and not where an image is. It also gives more flexibility for the future, if we need to refer to things other than images. Putting it in the image spec would also tie up the lifecycles, and I'm not sure that's a good thing.

We've muddied the definition, particularly with defining the usage for artifacts, and the image layout. The spec also uses "image" for a lot of things that aren't necessarily an image, like the image index.

[Jamstah]

Linking my thoughts on what the spec could include here: https://github.com/Jamstah/reference-spec/blob/main/spec.md

[mtrmac]

I think such a codification of existing practice would make sense. (I’m a tiny bit worried about a separate spec working group finding opportunities to invent new features with little previous precedent.)

---

https://github.com/Jamstah/reference-spec/blob/main/spec.md
Worth referencing different transports? Even if it's just to say they exist and are client specific?

(As the maintainer of the linked Skopeo tool), at best as a warning. Strings are notoriously ambiguous. IMHO every string field / syntax should strictly define whether it is an “image reference” in the sense of the rest of that text, or a transport:… string.

E.g. consider dir:foo. That could be Skopeo’s docker://docker.io/library/dir:foo when interpreted by Docker CLI, or it could be Skopeo’s dir:$(pwd)/foo when interpreted as a transport:… syntax. Hence Skopeo requires the user to always include a transport (OTOH Podman does rely on heuristics).

Accepting both syntaxes in the same field forces the consumer to resort to heuristics to disambiguate. And while it would certainly be more useful for every software to use the same heuristics, it seems to me to be very hard to design and standardize heuristics that will stand the test of time, as other registry features, and new c/image transports (or transports in other image codebases?), are added.

[jpbetz]

Kubernetes contributor here.

We would benefit significantly from having this spec. We're current dealing with a proposal to improve our validation support for image references (kubernetes/kubernetes#130834) and it's quite risky to offer guarantees on our API surface without an actual spec.

[sudo-bmitch]

Note that this PR is currently held in draft state waiting on a sufficient number of volunteers to be a proposed owners and stakeholders in the working group.

[Jamstah]

Do volunteers need any particular credential?

I'm happy to be involved.

[sudo-bmitch]

Do volunteers need any particular credential?

I'm happy to be involved.

Mostly a willingness to roll up their sleeves and do the work. Submit a review with your name/project to add if you have the bandwidth. See the referrers WG for an example of what it took to push through a large change.

The auth WG hit a wall when it came time to write the spec changes, and I still need to spend a lot of time doing the heavy lifting over there. I think we had more people interested in seeing the result than we had people with the time to lead the effort. So I'm holding off opening up new working groups where I'd end up being the only author.

[tianon]

See the referrers WG for an example of what it took to push through a large change.

The successfulness of that WG is also not universally held (I'd say there's at least as much cautionary tale there as blueprint, if not more).

[AkihiroSuda]

containerd, nerdctl, Moby, BuildKit can be added here?
cc @dmcgowan @tonistiigi @thaJeztah

[sudo-bmitch]

Is that a question to your co-maintainers or a request for me to add it?

[AkihiroSuda]

Both, but kinda latter one

[tianon]

The goal here is essentially to take the Grammar from https://pkg.go.dev/github.com/distribution/[email protected]#pkg-overview and write slightly more words around it, especially describing the docker.io, docker.io/library/, and :latest defaults, right?

(Perhaps to be even more clear, if this working group ends up designing something new, I think we've missed the mark.)

In other words, I'm not strictly opposed to this working group, but I don't think "extension" should be explicitly part of the initial charter. I also think this "spec" probably belongs in either the existing image-spec or distribution-spec. It's got a little bit of overlap between them, but that's the nature of those two specs already, and IMO this probably falls more on the distribution-spec than the image (as these "references" are describing objects as referenced via that API).

[tianon]

Adding a proper "review" so it's clear I'm -1 on WG this as-written (but +1 on the high-level intent of having spec-language for "references"); I've included some more explicit review comments here, but they are not totally inclusive (because I don't want to repeat myself too heavily).

[tianon]

"define" is a loaded word for me here, given the pre-existing nature of the relevant format -- would prefer a word like "document"

Also, "in the OCI" instead of "as an OCI spec" (because as I've noted prior, I don't think this should be new a spec, but declaring that it will be a new spec right off the bat in the WG proposal is definitely premature).

Suggested change

	This WG seeks to define the syntax and parsing of a reference as an OCI spec.
	This WG seeks to document the syntax and parsing of a reference within the OCI.

or

Suggested change

	This WG seeks to define the syntax and parsing of a reference as an OCI spec.
	This WG seeks to spec the syntax and parsing of a reference within the OCI.

[tianon]

This is all what I'd describe as "Referrers 2.0" (the cautionary tales I referenced earlier) - stretch goals that look more like designing a new thing than documenting an existing thing, and I'm pretty strongly opposed to including these in the first pass of explicit spec language for "references" in the OCI. Let's start smaller and just have an explicit spec for the thing that currently exists and has seen wide adoption for ~10+ years, then we can talk about expansion possibilities with less fog.

[mtrmac]

I agree here. There are many places that could refer to a spec documenting the existing state, that would be, for backward compatibility reasons with handling existing inputs, ~unable to adopt new many plausible new syntaxes (there are only so many ASCII characters, and for readability, some are more attractive than others, so conflicts are likely).

The list above is also unclear about the target user — some seem to be relevant to running containers, some to building them, some for, I guess, managing a registry, some for… I don’t know, guessing, purely internal uses within a build system/registry?

---

Philosophically, there are many data transfer / RPC mechanism that can be used to add such parameters to operations where the parameters make sense; and clear separation between data/metadata is essential. I strongly think that adding a field is almost always preferable to adding extra syntax within an existing string field.

The one exception is if the string field is so extremely widespread that adding a field everywhere we have the existing string is unfeasible. [E.g. we have ended up with the concept of a “slash-separated [0-terminated] file path” although that is, in many ways, flawed.] But, such a widespread use almost always implies that the specifics of the format are hard-coded in many places, and various extensions / other string formats assume that the format will not change.

---

I can imagine an effort to add features / drive a cross-vendor consensus for a single specific use case. E.g., maybe there is some feature that would be really valuable to have in K8s pod specs, and equivalent container image references in other “run a container from this image” orchestration systems. That is a reasonably bounded problem where clear benefits can be articulated.

I think add ing features that could benefit various sort-of-related but rather different use cases like (running / publishing / downloading / SBOMs / graph queries), where most of the proposed features would be irrelevant to any single user but many interoperability / backward compatibility concerns would be a concern for many such users, seems unlikely to succeed.

[sudo-bmitch]

Let's start smaller and just have an explicit spec for the thing that currently exists and has seen wide adoption for ~10+ years, then we can talk about expansion possibilities with less fog.

If the thing specified cannot be extended, there will be no "then". This isn't defining the possible extensions, just making it possible for extensions to be created. The result is likely limiting the character set for various fields so that characters are available to mark future extensions.

I strongly think that adding a field is almost always preferable to adding extra syntax within an existing string field.

I see a reference more as a URI than as a pointer to a tag on a registry. The tag pointer was already extended to allow for pointers to a digest. With referrers and people mounting OCI volumes, I can see a request for a reference to a referrer to an image. This is going to keep happening, e.g. with a pull through proxy that can proxy multiple registries, we will lose the ability to directly pull content from the proxy using the current reference syntax.

[tianon]

This increases my confusion -- we're trying to talk about "backwards compatibility" but we're explicitly trying to design a whole new thing?

[sudo-bmitch]

We're defining something that can be an in-place upgrade for tooling that currently uses the distribution/reference strings. v1.0 may end up being the current state of distribution/reference, but v* should avoid changes that break users who depend on the current string syntax. New features should be additive in a way that can be distinguished from current use cases, without ambiguity.

[tianon]

As noted above, I strongly disagree with this conclusion, but also believe it's premature to decide that this should be a completely net-new spec.

The goal here is essentially to take the Grammar from https://pkg.go.dev/github.com/distribution/[email protected]#pkg-overview and write slightly more words around it, especially describing the docker.io, docker.io/library/, and :latest defaults, right?

This isn't very big, and I don't think we should encourage it to be.

[sudo-bmitch]

I don't think the size of the work should determine whether it's a new spec, but rather whether it fits within the definition of an existing spec. Looking at image-spec:

The OCI Image Format project creates and maintains the software shipping container image format spec (OCI Image Format).

I don't think we fit there. And for distribution-spec:

The OCI Distribution Spec project defines an API protocol to facilitate and standardize the distribution of content.

We only fit there if we are deciding in advance to only allow references to content on a registry, and not to any other way of storing images, e.g. the OCI Image Layout.

[jonjohnsonjr]

In other words, I'm not strictly opposed to this working group, but I don't think "extension" should be explicitly part of the initial charter. I also think this "spec" probably belongs in either the existing image-spec or distribution-spec. It's got a little bit of overlap between them, but that's the nature of those two specs already, and IMO this probably falls more on the distribution-spec than the image (as these "references" are describing objects as referenced via that API).

I agree with this. It seems to me that a reasonable deliverable for the WG (if this really needs a WG) would be a PR against distribution-spec.

[sudo-bmitch]

I agree with this. It seems to me that a reasonable deliverable for the WG (if this really needs a WG) would be a PR against distribution-spec.

The community has gone back and forth on this a few times. It was originally proposed as a new spec, then there was a decision that it belonged as part of image-spec since it wasn't a registry API change, then it was back to making it a separate spec because image-spec should focus on the image json schemas and layer formats, and now it's over to distribution-spec where I'd suggest it prematurely limits it so that a reference to anything outside of a registry is permanently out of scope.

Define how the syntax can be extended to support other use cases

The goal with that section of the scope is to avoid defining a string that cannot be extended. There's a lot of external knowledge we can leverage for this, like the syntax of a URI, but that depends on ensuring character sets for various fields are limited from the start so that characters are available for future extensions. This doesn't mean those extensions need to be added as part of the working group, just that we acknowledge that groups will be doing this and to avoid rushing into a quick decision that locks us into a syntax that can never by extended without breaking changes.