Add release-process.md

[jonasnick]

This is an attempt at a simple release process. Fixes #856. To keep it simple, there is no concept of release candidates for now.

The release version is determined by semantic versioning of the API. Since it does not seem to be a lot of work, it is proper to also version the ABI with the libtool versioning system. This versioning scheme (semver API, libtool versioning ABI) follows the suggestion in the autotools mythbusters.

Experimental modules are a bit of a headache, as expected. This release process suggests to treat any change in experimental modules as backwards compatible. That way, users of stable modules are not bothered by frequent non-backwards compatible releases. But a downside is that one must not use experimental modules in shared libraries (which should be mentioned in the README?). It would be nice if we could make the schnorrsig module stable in the not too distant future (see also #817).

[rustyrussell]

This looks good! Note that c-lightning has a tool which extracts Changelog entries from commit messages, and also a bot that tests they exist for each PR (or Changelog-None is in GitHub description of PR).

[prusnak]

I suggest to use the format described here: https://keepachangelog.com/ - the guide also mentions that ChangeLog is not a git log dump :-)

[rustyrussell]

I suggest to use the format described here: https://keepachangelog.com/ - the guide also mentions that ChangeLog is not a git log dump :-)

Just to be clear: our tool extracts lives starting with Changelog-(Added|Fixed|Changed|Deprecated|Removed) and adds links to the PRs and sorts them :)

[prusnak]

our tool extracts lives

God, what kind of tools do you use?! 😁

[jonasnick]

Thanks for the feedback. I added an Unreleased section to the changelog and categories as suggested by keepachangelog.com. I think in most cases the titles of PRs are sufficiently descriptive such that CHANGELOG entries can be title of PR + link. EDIT: And if a PR fixes, adds or changes multiple distinct things then ideally they would be listed separately in the CHANGELOG.

[real-or-random]

ACK mod nit

[real-or-random]

ACK 56c5bff

[elichai]

ACK 56c5bff

[sipa]

ACK procedure/changes, but somehow CI does not like this.

[jonasnick]

rebased

[real-or-random]

utACK 86d0b9e

[sipa]

Hmm, what is the expected difference between the package version and the library version? Why would they not be the same?

[real-or-random]

@sipa See https://autotools.io/libtool/version.html

@jonasnick Sorry, needs trivial rebase again.

[sipa]

Can we write in the release process document how each of the 6 numbers are supposed to be updated as a function of the changes in such releases?

[sipa]

Another question (not necessarily something to be explicitly explained in the release process here, but still something that isn't currently clear to me): what branches structure are the release branches going to use? Possibilities:

• All release branches branch off master, and thus there is no (clean) way to create a x.y.(z+1) after x.(y+1).z has been released.
• The branch for x.(y+1).0 branches off the x.y.0 branch; the branch for x.y.(z+1) branches off the x.y.z one. Open questions with this:
  • How to obtain monotonically increasing library "revision" value in the presence of branches?
  • When creating a new branch by forking an already-released branch, would that imply resetting the IS_RELEASE variable? Or do we create the fork at the last commit before the one that sets the IS_RELEASE variable?

[jonasnick]

@sipa The rules are described in configure.ac ("The library version is based on libtool versioning of the ABI. The set of rules for updating the version can be found here: https://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html") or do you mean copying over the rules to the release-process.md? I thought it's best to have the rules where they version is set because it serves both as a doc of the existing values as well as how to update.

Another question

That's a good question. I'd like it answered in the release_process.md. How does Bitcoin Core do this? I would have thought everything is based off master except patches to earlier versions are done by branching off the earlier release branch. I think this would only have to be done for patches because it's unlikely we release a x.(y+1).z when (x+1).y.z is already released. Btw, "revision" isn't monotonically increasing, it's set to if there's any change to the interface (including additions).

[sipa]

@jonasnick Yes, I meant including the actual rules for incrementing the library version somewhere, not just a reference. I find the rules described in that link kind of vague, and I also had never heard about the concept of library versions at all; if someone is in a similar situation, and just sees the link they may not realize it's actually explaining why it's different from the numbers above.

That's a good question. I'd like it answered in the release_process.md. How does Bitcoin Core do this? I would have thought everything is based off master except patches to earlier versions are done by branching off the earlier release branch. I think this would only have to be done for patches because it's unlikely we release a x.(y+1).z when (x+1).y.z is already released. Btw, "revision" isn't monotonically increasing, it's set to if there's any change to the interface (including additions).

Bitcoin Core's release process is roughly:

• As a general rule, with very few exceptions, all PRs are made to the master branch.
• Every so often the master branch is forked off to create a major release branch (e.g. the 22.x branch). After that point, future changes in master may be backported to the 22.x branch if they're bugfixes or somehow specifically relevant for that release. At some point, release candidates v22.0rc1, v22.0rc2, ... are created from this branch, and finally the final v22.0 tag is created.
• Minor releases (e.g. v22.1) are created by continuing to backport changes (bugfixes, mostly) from the master branch into the 22.x branch after v22.0 is released, en then later in the same branch tagging v22.1rc1, v22.1rc2, ..., and v22.1. These can be made for multiple major release branches simultnaneously (e.g. a bugfix in master is sufficiently important that it deserves going into major releases 21 and 20 too, could be backported to all these branches, and then trigger release candidates/releases of new minor releases in all of them.

So the major difference with what you're proposing is that Bitcoin Core only has branches for major releases, not for very release, and that branch is then used for potentially multiple tags.

Btw, "revision" isn't monotonically increasing, it's set to if there's any change to the interface (including additions).

It's set to what if there is any change to the interface?

[sipa]

@jonasnick Oh, I noticed that the link in the source is different from the one @real-or-random linked to. That one gives much more concrete instructions already.

[jonasnick]

I don't see signficant down or upsides with either branching approach so I'd follow Bitcoin Core's process where applicable because that was the initial intention anyway.

It's set to what if there is any change to the interface?

Lol, sorry, I have a new (terrible) keyboard. 0.

[sipa]

Well, I think there are probably some differences that could justify deviating. For one, I don't think we expect many "major" releases (in the semver sense) after 1.0, certainly not two per year.

[apoelstra]

utACK 3ed0d02

Looks great!

[real-or-random]

ACK 3ed0d02

[elichai]

ACK b4b02fd

[real-or-random]

@elichai wrong commit :P

[elichai]

@real-or-random Ops, somehow I keep getting confused on the order of commits in github (in git the latest is on top, in github it's at the bottom)

ACK 3ed0d02

[apoelstra]

@elichai if you have

[remote "origin"]
        fetch = +refs/pull/*:refs/remotes/pr/*

in your gitconfig, you can do something like git rev-parse pr/964/head | xsel to get the top commit ID.

You may still type the wrong PR number or write "ACK" in the wrong browser tab, but at least there's one less mistake you can make :)

[sipa]

ACK 3ed0d02