Consider publishing grpc-gateway to Bazel Central Registry

[hharrisd]

We maintain an SDK that depends on grpc-gateway. We're encountering challenges with Bazel's module system, as override directives only work in root modules.

Publishing grpc-gateway to BCR would:

• Eliminate the need for overrides
• Ensure consistent builds across all dependent projects
• Reduce maintenance overhead for downstream users
• Align with Bazel's recommended dependency management practices

We're willing to help with the BCR publication process if you're interested. Would you consider this addition to improve the experience for grpc-gateway users?

[johanbrandhorst]

Thanks for your issue. We've talked about doing this before so it's definitely something I'd consider. Please feel free to open a PR. I want to make sure this doesn't unduly add to the maintenance burden of this repository, so I'd want publishing to the registry to happen automatically when a new release is built. I'm happy to registry any accounts and populate secrets necessary for CI. CC @AlejoAsd who helped add our bzlmod support.

[hharrisd]

Hi @johanbrandhorst, thank you for your answer.

I have been reading about publishing modules in BCR. The simplest and most recommended method is to use Publish to BCR.

Process description

Publish to BCR is a GitHub app that mirrors releases of your Bazel ruleset to the Bazel Central Registry.

The goal is that when a grpc-gateway version is released, the app will create a PR in a forked BCR repository.

Initially, the released version is sanity-checked: the app runs a presubmit job, which builds and runs tests on module targets. If the tests are successful, a PR is built for the BCR.

IMPORTANT: running the presubmit job will require CI credits. Are you OK with adding this extra pipeline?

Process diagram

Configuration Steps

1. Install and configure Publish to BCR for two repositories.
   • This repository grpc-gateway.
   • A fork of bazelbuild/bazel-central-registry.
   • We need from the grpc-gateway team to proceed with this configuration.
2. Add templates to construct a new BCR PR after a release.
   • I'm working on these templates. I will create a PR for this and add @AlejoAsd as reviewer.
   • In this step, we need the grpc-gateway team to provide the maintainer's data (name, email, and GitHub user).
3. Cut a release. A pull request will be opened against the BCR.

The steps are described in the Publish to BCR repo.

[johanbrandhorst]

Thanks for looking into this. The process sounds good to me - just one concern. Can we run the presubmit as part of our CI, to ensure there's no risk of making a release that can't pass the presubmit and make it to the BCR? I would hate to have to re-release something just because it failed the presubmit.

Where would the bazel-central-registry fork live? I can create it in grpc-ecosystem, but is it a necessity? I'm happy to volunteer as the project maintainer for the purpose of creating the PRs against a user.

[hharrisd]

Thank you for your answer.

Regarding your questions:

1. Can we run the presubmit as part of our CI, to ensure there's no risk of making a release that can't pass the presubmit and make it to the BCR?

   Presubmit should include essential build and test targets used to verify a module version. For instance, I propose running tests in your root and examples folders targets.

   Likewise, a set of validations is checked during presubmit. A tool in the BCR repo can run those presubmit validations locally. Maybe it could be integrated into your CI.

2. Where would the bazel-central-registry fork live? I can create it in grpc-ecosystem, but is it a necessity?

   According to Publish to BCR:

   A fork of bazelbuild/bazel-central-registry. The fork can be in the same GitHub account as your ruleset or in the release author's personal account. If you use release automation and the release author is the github-actions bot, then the fork must be in ruleset's account unless you override the releaser.

[johanbrandhorst]

Cool, I think our existing Bazel CI jobs probably already satisfy the presubmit requirements. I'm also happy to create the BCR fork in this org, maybe it can serve other repositories in this org too.

[hharrisd]

Hi Johan, I hope you are doing great.

I've been working on adding the required files for the publish_to_bcr plugin, and I think we're in good shape.

One thing that we are facing is that we need to add an asset to the release that contains a repo archive. It has the exact content of the source archive assets generated by the release process but with a different checksum and link structure. This situation is explained in this article.

Here is an example of the release source I generated and uploaded manually for a fork of this repo.

The release archive could be generated with the command git archive. I noticed that grpc-gateway uses GoReleaser to manage the release process, so I think that we could add a step to generate the release archive:

In the release.yml file, I could add a new step to create and upload the release source file:

  - name: Create source archive
    if: startsWith(github.ref, 'refs/tags/')
    run: |
      REPO_NAME=$(echo "$GITHUB_REPOSITORY" | cut -d '/' -f 2)
      TAG_NAME=${GITHUB_REF#refs/tags/}
      
      # Create the archive
      git archive --format=tar.gz --prefix=${REPO_NAME}-${TAG_NAME}/ ${TAG_NAME} > ${REPO_NAME}-${TAG_NAME}.tar.gz
      
      # Upload to release
      gh release upload ${TAG_NAME} ${REPO_NAME}-${TAG_NAME}.tar.gz
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

That being said, is generating this archive as part of your release assets feasible?

Second, I don't have experience with GoReleaser, so I'd like to ask if you consider this a proper way to generate the archive source asset, or do you have a different suggestion?

If you allow me to proceed with it, I could create a different PR for the release archive work.

CC: @AlejoAsd

[johanbrandhorst]

Feasible? Sure. I haven't looked into whether using gorelease specifically would be better, but note that we generate checksums and an auditible provenance file, so if we add a separate source archive (zip? tar?) we should probably include that in the generate checksum files. Please do feel free to experiment with a PR, you can temporarily change the release workflow trigger to test run it (though I think you may need to set a goreleaser test run flag or something).

Also, I got an email saying something like

Failed to publish entry for hharrisd/[email protected] to the Bazel Central Registry.

Received status 404

Should I be worried about this?

[hharrisd]

Thank you, Johan.

I'll work on the release process. I noticed the checksum generation, which is why I think using GoReleaser should be the smoother option to keep the provenance work.

Yes, I received the email as well. It was sent when I created a test release in the fork. I wrote in the Bazel Slack to see if there is a way to test the integration of the publish_to_bcr plugin without creating a release, but it seems there isn't another way yet.