What's the versioning strategy/guarantees?

[znewman01]

I think we should document guarantees around versioning. It's not clear what changes are/aren't breaking (to the CLI vs. library API), and if they are breaking, what the process for making them is.

This discussion is a good venue to bring up ideas/questions. Hopefully at the end we can codify it in a doc. Call it, say VERSIONING.md. It will include the resolutions of most of the questions in this discussion. This will be useful for code reviewers as well as users of the API and CLI.

EDIT: there is existing documentation about versioning, but it leaves most of the below questions unaddressed.

EDIT EDIT: Proposal: Cosign Versioning aggregates many of the below discussions into one place so we can consider adopting a new versioning policy!

[znewman01]

Proposal: decouple CLI versioning and library versioning.

The CLI and library have different users. We could make a breaking change to one without affecting the other--for instance, removing an unused method.

I think that we should stop versioning them in tandem, because the versions track different things.

This will positively improve release notes, but require some changes to release processes.

[hectorj2f]

I'll suggest following the policy around Kubernetes versioning as a guidance on how to handle them here.

[znewman01]

Do you have a reference? I should have asked that 2 weeks ago, haha.

[znewman01]

Question: how long are old CLI versions supported?

The Sigstore infrastructure might change in a way that's ambiguous as to whether it's breaking or not. If an old Cosign version was doing something incorrectly, how long should we support that behavior server-side?

(This is slightly off-topic but feels relevant enough I'd like to discuss it.)

[asraa]

I would expect something like 1 year in time for major releases, and that there should be new releases available during that 1 year deprecation window that users can take to migrate to.

I think it's relevant for patching. Some incorrect behavior is going to be unavoidable (CVEs, security errors), and those old Cosign versions MUST be updated to a new patch version.

I expect we would need something like a regular release cadence though to support this.

And some way (tests) to ensure that this policy is enforced (or regression testing all clients, not just cosign for 1 year support)

[znewman01]

Question: what's a breaking change in the CLI?

For instance, which of these count? Some are obvious, some less so. See Hyrum's Law and XKCD #1172.

• A change to behavior gated on COSIGN_EXPERIMENTAL
• A change to how we interpret an environment variable
  • What if it's undocumented?
  • What if it's labeled "for testing only"?
• A change to a flag name
• A change to a message printed on STDOUT
  • What if we're adding a line?
  • What if the user has requested a specific output format (JSON)?
  • What if we're adding a field in a machine-readable output?
• A change to a message printed on STDERR
• Requiring user input where none was required before
• Something that used to verify successfully no longer does.

Can we come up with a guiding principle that's easy to explain? For instance (not advocating for anything in particular):

• If it breaks your script, it's a breaking change.
• If it breaks your script AND you're following best practices X and Y, it's a breaking change except changes to W and Z.
• If a human's workflow breaks, it's a breaking change.

[znewman01]

I once worked on a CLI that got so tired of answering these questions that they just started bumping the major version every time they did a release. That's somewhat user hostile so would be nice to avoid doing that here, but illustrates the trickiness.

[znewman01]

Question: what's a breaking change in the API?

Obviously, removing a public method counts. But what if it's a method we're pretty sure nobody uses? What if the method never worked in the first place? What about a public method in the cmd/ package? (We've been treating that implicitly as unversioned.)

What about changes to behavior? What if the behavior was a bug? What's a bug vs. a feature? What if the behavior was undocumented? What if the whole API was undocumented---does that give us license to change everything?

[imjasonh]

My view is the Go APIs in cosign have no compatibility guarantee. We should document this expectation, and not casually/gleefully break folks, but otherwise deprecating and removing things is right and good.

We should strive to provide a compatibility guarantee for sigstore-go, when it exists (and document it).

After sigstore-go exists, if there's still code in cosign that we expect folks to consume as a Go library -- likely stuff around interpreting OCI stuff as signatures/SBOMs/etc., once OCI APIs have moved to go-containerregistry -- we should document the compatibility expectations.

In any case I think there's value to having APIs that move quickly while they need to, then add compatibility guarantees only after the pace of changes has slowed down. Enforcing compatibility guarantees too early makes it hard to make changes to improve APIs, fix bugs, etc. Go lets you pin to a specific version if you need to use an old API.

[znewman01]

In any case I think there's value to having APIs that move quickly while they need to, then add compatibility guarantees only after the pace of changes has slowed down.

Yeah! Maybe you could indicate that the pace of changes has slowed down with a 1.0 release? 😛

Too bad nobody asked me before we did that. I would love to call a mulligan though and do what you're proposing. We'll see how much pain that would cause.

[znewman01]

Proposal: broad exception for security issues

Some security issues require "breaking" fixes. For instance, if something verifies successfully but it shouldn't, it could be considered breaking to fix that.

IMO, it's pretty clear that we want to do that, and we don't want a major version bump because that involves friction.

[znewman01]

Proposal: follow semver pretty strictly for the library

As a Go library, we are supposed to follow the idiosyncratic beliefs of the Go core team about versioning: namely, that perfect semantic versioning is possible and that "what is the public API of this package" is a question that can be answered unambiguously, and no exceptions are allowed unless you're really special. I'll leave it as an exercise to the reader to figure out whether I think this is a wise policy, but these are the rules regardless and IMO we should follow them, especially because Go's build tooling assumes that we do. Otherwise, we should very explicitly document that we don't.

[znewman01]

Proposal: have an "experimental" section

Now that Cosign is v1, every new addition is technically supposed to be permanent.

That's annoying---it doesn't give any lattitude to experiment with APIs. Should
we add a pkg/experimental/ section of the codebase where all semver bets are
off?

[znewman01]

Question: how to track things that we're waiting on a new major version to fix?

There are things in the Cosign API that we're planning to remove/change in v2 (example).

A lot of these require very straightforward code changes. But it'd be annoying to push a major version and forget to do that. How can we make sure we don't forget?

Some ideas related to this:

• In GitHub: a tag for issues, or a milestone
• In code: use a // DEPRECATED: or // PLANNED CHANGE: comment in the docs
• Require that the work be done up front. So if you want your breaking change, you should gate it behind a (non-public!) flag that can be flipped.

This is basically a question about how we should do deprecation.

[znewman01]

Proposal: We should issue a v2 sooner rather than later.

I frequently notice a weird aversion to publishing new major versions of a library. While I think it's a good idea to weigh the benefits of a proposed change against the cost to users forced to update their calling code and to not make breaking changes willy-nilly, it can get a little silly if you avoid making a good/useful change to the library to avoid breaking an interface that probably 0 people use.

I think the v2 release has the largest psychological overhead and propose we should do it sooner rather than later. After that point, I think such decisions will be made a little more rationally.

[znewman01]

See also: #2376

[znewman01]

Proposal: use automation to detect breaking changes in PRs

It's pretty easy to miss breaking changes while reviewing code (I've done it).

Proposal: add automation, like gorelease, to detect these in PRs.

[bennyvasquez]

In the community meeting this week @znewman01 shared this versioning proposal for Cosign. I wanted to make sure that got shared here, too.

He'd mentioned he was looking for feedback by the end of this week, but can we extend that a little, @znewman01? Maybe just another week (or two, given the US holiday next week), so folks can engage if they care to.

[znewman01]

We can wait until the week of Nov 28th to make any changes to policy here

[rmueller83]

The versioning of the container packages seems broken to me. The latest tag points to the recently released v1.13.2 where it previously pointed to v2.2.1 (which is still marked as Latest in the releases section).
Also, it would be nice if there was a v2 container tag which points to the latest v2.x.x version.

[haydentherapper]

@cpanato I think we accidentally changed latest with the release of 1.13.2.

[haydentherapper]

@rmueller83 thanks for catching this, latest has been updated to 2.2.1. please let me know if you run into this again