Toolchainize //scala_proto:{,deps_}toolchain_type #1687

mbland · 2025-01-27T15:59:17Z

Description

Adds scala_proto toolchains to scala_toolchains(). Part of #1482.

The most significant part of the change is moving all the toolchain rules from scala_proto/BUILD to setup_scala_toolchains() in scala_proto/toolchains.bzl.

Adds the scala_proto_deps_providers() macro to replace //scala_proto:scalapb_{compile,grpc,worker}_deps_provider targets in the dep_providers parameter of scala_proto_deps_toolchain(). Examples of this are in test/proto/custom_generator/BUILD.

Excludes @scala_proto_rules_scalapb_protoc_gen from DEFAULT_SCALAPB_WORKER_DEPS in scala_proto/default/default_deps.bzl for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended.

A lot of the other changes are more opportunistic removals of @io_bazel_rules_scala label prefixes and application of Label() where appropriate.

Motivation

This is the next step in the toolchainization of the frameworks used to implement various rules, as required to implement Bzlmod support.

Excluding @scala_proto_rules_scalapb_protoc_gen from DEFAULT_SCALAPB_WORKER_DEPS avoids build failures under Bzlmod, since:

This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8.
Importing the nonexistent scala_proto_rules_scalapb_protoc_gen under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions.
MODULE.bazel can iterate over a list of Scala versions, filtering out Scala 2.11, and call use_repo() on each version specific repo.

As in other pull requests, removing @io_bazel_rules_scala label prefixes will allow Bzlmod users to use rules_scala without setting repo_name = "@io_bazel_rules_scala" in bazel_dep().

Adds scala_proto toolchains to `scala_toolchains()`. Part of bazel-contrib#1482. The most significant part of the change is moving all the toolchain rules from `scala_proto/BUILD` to `setup_scala_toolchains()` in `scala_proto/toolchains.bzl`. Adds the `scala_proto_deps_providers()` macro to replace `//scala_proto:scalapb_{compile,grpc,worker}_deps_provider` targets in the `dep_providers` parameter of `scala_proto_deps_toolchain()`. Examples of this are in `test/proto/custom_generator/BUILD`. Excludes `@scala_proto_rules_scalapb_protoc_gen` from `DEFAULT_SCALAPB_WORKER_DEPS` in `scala_proto/default/default_deps.bzl` for Scala 2.11. For other Scala versions, this repo name will have the Scala version appended. This is to avoid build failures under Bzlmod, since: - This repo is required by ScalaPB 0.11.17, but Scala 2.11 is capped at ScalaPB 0.9.8. - Importing the nonexistent `scala_proto_rules_scalapb_protoc_gen` under Scala 2.11 results in an error under Bzlmod, as does importing it multiple times when configuring multiple Scala versions. - `MODULE.bazel` can iterate over a list of Scala versions, filtering out Scala 2.11, and call `use_repo()` on each version specific repo. A lot of the other changes are more opportunistic removals of `@io_bazel_rules_scala` label prefixes and application of `Label()` where appropriate. Doing this will allow Bzlmod users to use `rules_scala` without setting `repo_name = "@io_bazel_rules_scala"` in `bazel_dep()`.

mbland · 2025-02-03T16:40:20Z

Rebased after #1688 and #1689.

simuons

Thanks, @mbland!

@dmivankov

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0. Part of bazel-contrib#1482. Closes bazel-contrib#1652. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `abseil-cpp`: 20220623.1 => 20250127.0 - `rules_java`: 7.12.4 => 8.9.0 - `protobuf`: 21.7 => 29.3 - `rules_proto`: 6.0.2 => 7.1.0 - `google-common-protos`: 2.52.0 => 2.53.0 This precipitated the following updates also included in this commit: - Loads `java_proto_library` from `com_google_protobuf`. - Bumps `.bazelversion` to 7.5.0. - Sets `.bazelci/presubmit.yml` to use Bazel 7.5.0 instead of 6.5.0, and `last_rc` in place of `7.x`. - Sets `common --enable_workspace --noenable_bzlmod` in `.bazelrc` and `tools/bazel\.rc\.buildkite`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases. Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. `WORKSPACE` builds succeed under Bazel 6.5.0 (with C++ compiler flags) for the time being, but are not guaranteed to continue working. (Bazel 6 Bzlmod builds would break anyway, because Bazel 6 doesn't provide `use_repo_rule`, required by `rules_jvm_external` 6.3, which is required by `protobuf` v29.) The `README` now documents that `scala_proto` or any rules otherwise depending on `protobuf` are no longer supported out of the box for Scala 2.11. Such users will have to ensure they register their own downgraded versions, at their own risk of future `rules_scala` 7.x incompatibility. Version bump related updates to `.bazelversion`,`WORKSPACE`, and `third_party/repositories` comprise the bulk of this change. Other small, yet important changes (other than those due to the version bumps noted above) include: - Adding a new `examples/overridden_artifacts` repository and the `overridden_artifacts_example` test case in `test/shell/test_examples.sh`. - Making `_validate_scalac_srcjar()` and `dt_patched_compiler_setup()` in `scala/private/macros/scala_repositories.bzl` more tolerant of dictionaries containing keys mapped to `None`. The new `overridden_artifacts_example` test covers this. Bazel 8 and `rules_java` 8 require `protobuf` >= v29. After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Here's why the other changes were necessary in light of the version bumps: - `java_proto_library` from `rules_java` is now officially deprecated, hence loading it from `com_google_protobuf`. - Setting `common --enable_workspace --noenable_bzlmod` (instead of `build`) fixes `test_semanticdb_handles_removed_sourcefiles`. This test relies on `bazel query`, which is also affected by these flags, hence `common` instead of `build`. Bazel 8 defaults to `--enable_bzlmod --noenable_workspace`, causing the `WORKSPACE` run of this test to fail. - `glob` requires an explicit `allow_empty = True` parameter now that `--incompatible_disallow_empty_glob` defaults to `True` in Bazel 8. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` >= 26.0. For this reason, we must remove the Scala 2.11 test cases. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688 Finally, the motivation for the non-version bump changes: - The design bug in the upcoming Bzlmod API for `overridden_artifacts` reported by @dmivankov precipitated the `examples/overriden_artifacts` repo and test. See: bazel-contrib#1482 (comment) bazel-contrib#1482 (comment) - The `_validate_scalac_srcjar()` update maintains the strict checks for mutually exclusive values, while preventing client code from having to explicitly filter out `None` entries. This pairs with the change in `dt_patched_compiler_setup()` that uses the `compiler_srcjar` dictionary. These changes helps keep the upcoming module extension a bit cleaner.

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0. Part of bazel-contrib#1482. Closes bazel-contrib#1652. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `abseil-cpp`: 20220623.1 => 20250127.0 - `rules_java`: 7.12.4 => 8.9.0 - `protobuf`: 21.7 => 29.3 - `rules_proto`: 6.0.2 => 7.1.0 - `google-common-protos`: 2.52.0 => 2.53.0 This precipitated the following updates also included in this commit: - Loads `java_proto_library` from `com_google_protobuf`. - Bumps `.bazelversion` to 7.5.0. - Sets `.bazelci/presubmit.yml` to use Bazel 7.5.0 instead of 6.5.0, and `last_rc` in place of `7.x`. - Sets `common --enable_workspace --noenable_bzlmod` in `.bazelrc` and `tools/bazel\.rc\.buildkite`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases. Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. `WORKSPACE` builds succeed under Bazel 6.5.0 (with C++ compiler flags) for the time being, but are not guaranteed to continue working. (Bazel 6 Bzlmod builds would break anyway, because Bazel 6 doesn't provide `use_repo_rule`, required by `rules_jvm_external` 6.3, which is required by `protobuf` v29.) The `README` now documents that `scala_proto` or any rules otherwise depending on `protobuf` are no longer supported out of the box for Scala 2.11. Such users will have to ensure they register their own downgraded versions, at their own risk of future `rules_scala` 7.x incompatibility. Version bump related updates to `.bazelversion`,`WORKSPACE`, and `third_party/repositories` comprise the bulk of this change. Other small, yet important changes (other than those due to the version bumps noted above) include: - Adding a new `examples/overridden_artifacts` repository and the `overridden_artifacts_example` test case in `test/shell/test_examples.sh`. - Making `_validate_scalac_srcjar()` and `dt_patched_compiler_setup()` in `scala/private/macros/scala_repositories.bzl` more tolerant of dictionaries containing keys mapped to `None`. The new `overridden_artifacts_example` test covers this. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (the first supporting with Bazel 6, the second dropping support). Bazel 8 and `rules_java` 8 require `protobuf` >= v29. After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Here's why the other changes were necessary in light of the version bumps: - `java_proto_library` from `rules_java` is now officially deprecated, hence loading it from `com_google_protobuf`. - Setting `common --enable_workspace --noenable_bzlmod` (instead of `build`) fixes `test_semanticdb_handles_removed_sourcefiles`. This test relies on `bazel query`, which is also affected by these flags, hence `common` instead of `build`. Bazel 8 defaults to `--enable_bzlmod --noenable_workspace`, causing the `WORKSPACE` run of this test to fail. - `glob` requires an explicit `allow_empty = True` parameter now that `--incompatible_disallow_empty_glob` defaults to `True` in Bazel 8. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` >= 26.0. For this reason, we must remove the Scala 2.11 test cases. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688 Finally, the motivation for the non-version bump changes: - The design bug in the upcoming Bzlmod API for `overridden_artifacts` reported by @dmivankov precipitated the `examples/overriden_artifacts` repo and test. See: bazel-contrib#1482 (comment) bazel-contrib#1482 (comment) - The `_validate_scalac_srcjar()` update maintains the strict checks for mutually exclusive values, while preventing client code from having to explicitly filter out `None` entries. This pairs with the change in `dt_patched_compiler_setup()` that uses the `compiler_srcjar` dictionary. These changes helps keep the upcoming module extension a bit cleaner.

It turns out `protobuf` v26.6 isn't compatibile with ScalaPB 0.9.8 used by Scala 2.11: ```txt Caused by: java.lang.UnsupportedOperationException: As of 2022/09/29 (release 21.7) makeExtensionsImmutable should not be called from protobuf gencode. If you are seeing this message, your gencode is vulnerable to a denial of service attack. You should regenerate your code using protobuf 25.6 or later. Use the latest version that meets your needs. However, if you understand the risks and wish to continue with vulnerable gencode, you can set the system property `-Dcom.google.protobuf.use_unsafe_pre22_gencode` on the command line. See security vulnerability: GHSA-h4h5-3hr4-j3g2 ``` As mentioned in bazel-contrib#1710, we should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688

Contains many editorial improvements to and some extra information in the README, along with a few small improvements to the code. Broken out from bazel-contrib#1710, and part of bazel-contrib#1482 and bazel-contrib#1652. Specifically: - Adds more info on translating `@rules_scala` to `@io_bazel_rules_scala` for dependencies via repo mapping. - Merges information about the previously planned `rules_scala` 8.0.0 release into the information for 7.0.0, since it seems we may make only one major release. - Improves information about `protobuf` support for versions before v28, Scala 2.11, and the upcoming Bzlmod `compatibility_level` setting. - In `scala_config.bzl`, changes the private `_default_scala_version()` to the public `DEFAULT_SCALA_VERSION`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases from `test_thirdparty_version.sh` and `test_version.sh`. This change is smaller and more focused than bazel-contrib#1710, and should ultimately make that pull request smaller and/or easier to review. The motivations for the individual changes are: - The public `DEFAULT_SCALA_VERSION` constant makes this value accessible to the upcoming module extension. - `glob` requires an explicit `allow_empty = True` parameter in Bazel 8, in which `--incompatible_disallow_empty_glob` defaults to `True`. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` v25.6 or later. For this reason, we must remove the Scala 2.11 test cases, as documented in the `README.md` updates. See also bazel-contrib#1712. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688

* 7.0.0 README updates and minor code changes Contains many editorial improvements to and some extra information in the README, along with a few small improvements to the code. Broken out from #1710, and part of #1482 and #1652. Specifically: - Adds more info on translating `@rules_scala` to `@io_bazel_rules_scala` for dependencies via repo mapping. - Merges information about the previously planned `rules_scala` 8.0.0 release into the information for 7.0.0, since it seems we may make only one major release. - Improves information about `protobuf` support for versions before v28, Scala 2.11, and the upcoming Bzlmod `compatibility_level` setting. - In `scala_config.bzl`, changes the private `_default_scala_version()` to the public `DEFAULT_SCALA_VERSION`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases from `test_thirdparty_version.sh` and `test_version.sh`. This change is smaller and more focused than #1710, and should ultimately make that pull request smaller and/or easier to review. The motivations for the individual changes are: - The public `DEFAULT_SCALA_VERSION` constant makes this value accessible to the upcoming module extension. - `glob` requires an explicit `allow_empty = True` parameter in Bazel 8, in which `--incompatible_disallow_empty_glob` defaults to `True`. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` v25.6 or later. For this reason, we must remove the Scala 2.11 test cases, as documented in the `README.md` updates. See also #1712. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - #1631 - #1648 - #1687 - #1688 * Tweak `rules_python` and Bazel 6.5.0 `README` info Provided a better explanation for using `rules_python` 0.38.0 for now. Improved some of the language in the Bazel 6.5.0 compatibility section.

mbland requested review from liucijus and simuons as code owners January 27, 2025 15:59

mbland mentioned this pull request Jan 28, 2025

Support Bzlmod and add rules_scala to bazel-central-registry #1482

Closed

mbland force-pushed the bzlmod-toolchain-scala-proto branch from 7df3082 to 62c17f1 Compare February 3, 2025 15:04

simuons approved these changes Feb 4, 2025

View reviewed changes

mbland mentioned this pull request Feb 4, 2025

New repository_ctx.original_name produces "canonical reproducible" warnings bazelbuild/bazel#25192

Closed

liucijus approved these changes Feb 5, 2025

View reviewed changes

liucijus merged commit d409cb6 into bazel-contrib:master Feb 5, 2025
2 checks passed

mbland deleted the bzlmod-toolchain-scala-proto branch February 5, 2025 15:45

mbland mentioned this pull request Feb 28, 2025

Bazel 8 + rules_java 8 updates, protoc toolchain #1710

Merged

mbland mentioned this pull request Mar 7, 2025

7.0.0 README updates and minor code changes #1714

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Toolchainize //scala_proto:{,deps_}toolchain_type #1687

Toolchainize //scala_proto:{,deps_}toolchain_type #1687

mbland commented Jan 27, 2025

mbland commented Feb 3, 2025

simuons left a comment

Toolchainize //scala_proto:{,deps_}toolchain_type #1687

Toolchainize //scala_proto:{,deps_}toolchain_type #1687

Conversation

mbland commented Jan 27, 2025

Description

Motivation

mbland commented Feb 3, 2025

simuons left a comment

Choose a reason for hiding this comment