Linking errors with LTO on Ubuntu 24.04 and rules_rust 0.60.0

[bitemyapp]

.bazelrc:

build --compilation_mode=opt
build --@rules_rust//rust/settings:lto=thin
build --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cincremental=false

MODULE.bazel:

bazel_dep(name = "rules_rust", version = "0.60.0")

rust = use_extension("@rules_rust//rust:extensions.bzl", "rust")
rust.toolchain(
    edition = "2021",
    versions = ["1.86.0"],
)
use_repo(rust, "rust_toolchains")

register_toolchains("@rust_toolchains//:all")

on an Ubuntu 24.04 machine with LLVM 18 and 19 installed yields this error:

-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs" "-shared" "-fuse-ld=lld" "-B/usr/bin" "-Wl,-no-as-needed" "-Wl,-z,relro,-z,now" "-pass-exit-codes" "-Wl,--gc-sections" "-Wl,--push-state,-as-needed" "-lstdc++" "-Wl,--pop-state" "-Wl,--push-state,-as-needed" "-lm" "-Wl,--pop-state"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: ld.lld: error: /home/gh-runner/.cache/bazel/_bazel_gh-runner/58b7b47ee097fdc1b06b3e6b39197a42/execroot/_main/bazel-out/k8-opt/bin/external/rules_rust++crate+crates__syn-1.0.109/libsyn-912657071.rlib(syn-912657071.1khmagf21y6tm6h2dkwmm30ld.rcgu.o):
Invalid attribute group entry (Producer: 'LLVM19.1.7-rust-1.86.0-stable' Reader: 'LLVM 18.1.3')
          collect2: error: ld returned 1 exit status

Trying to set it to rust-lld expressly for linux:

build:linux --@rules_rust//rust/settings:extra_rustc_flag=-Clinker=rust-lld

yields this error:

"--gc-sections" "-shared" "-z" "relro" "-z" "now" "-O1" "--strip-debug" "-shared" "-fuse-ld=lld" "-B/usr/bin" "-Wl,-no-as-needed" "-Wl,-z,relro,-z,now" "-pass-exit-codes" "-Wl,--gc-sections" "-Wl,--push-state,-as-needed" "-lstdc++" "-Wl,--pop-state" "-Wl,--push-state,-as-needed" "-lm" "-Wl,--pop-state"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: unknown argument '-fuse-ld=lld'
          rust-lld: error: unknown argument '-B/usr/bin'
          rust-lld: error: unknown argument '-Wl,-no-as-needed'
          rust-lld: error: unknown argument '-Wl,-z,relro,-z,now'
          rust-lld: error: unknown argument '-pass-exit-codes'
          rust-lld: error: unknown argument '-Wl,--gc-sections'
          rust-lld: error: unknown argument '-Wl,--push-state,-as-needed'
          rust-lld: error: unknown argument '-Wl,--pop-state'
          rust-lld: error: unknown argument '-Wl,--push-state,-as-needed'
          rust-lld: error: unknown argument '-Wl,--pop-state'
          rust-lld: error: unable to find library -lgcc_s
          rust-lld: error: unable to find library -lutil
          rust-lld: error: unable to find library -lrt
          rust-lld: error: unable to find library -lpthread
          rust-lld: error: unable to find library -lm
          rust-lld: error: unable to find library -ldl
          rust-lld: error: unable to find library -lc
          rust-lld: error: unable to find library -lstdc++
          rust-lld: error: unable to find library -lm

Then I tried:

build:linux --@rules_rust//rust/settings:experimental_use_cc_common_link=True
build:linux --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-fuse-ld=lld

which got me:

Invalid attribute group entry (Producer: 'LLVM19.1.7-rust-1.86.0-stable' Reader: 'LLVM 18.1.3')

[bitemyapp]

build:linux --@rules_rust//rust/settings:experimental_use_cc_common_link=True
build:linux --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-fuse-ld=lld-19

Got me:

INFO: Analyzed 56 targets (0 packages loaded, 24983 targets configured).
ERROR: /home/gh-runner/.cache/bazel/_bazel_gh-runner/58b7b47ee097fdc1b06b3e6b39197a42/external/rules_rust++crate+crates__curve25519-dalek-4.1.3/BUILD.bazel:109:19:
Linking external/rules_rust++crate+crates__curve25519-dalek-4.1.3/_bs_ [for tool] failed: (Exit 1): gcc failed: error executing CppLink command (from target @@rules_rust++crate+crates__curve25519-dalek-4.1.3//:_bs_) /usr/bin/gcc @bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__curve25519-dalek-4.1.3/_bs_-0.params

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
collect2: fatal error: cannot find 'ld'
compilation terminated.

[bitemyapp]

with this:

build --compilation_mode=opt
build --@rules_rust//rust/settings:lto=thin
build --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cincremental=false
build:linux --@rules_rust//rust/settings:experimental_use_cc_common_link=True
build:linux --linkopt=-fuse-ld=lld

I get further but then one of the workspace crates fails without any obvious rustc errors (yes, I used verbose_failures and sandbox_debug), just says the object file is missing:

ERROR: /home/gh-runner/workspace/crate-name/BUILD.bazel:6:12: output 'crate-name/crate-name.o' was not created
ERROR: /home/gh-runner/workspace/crate-name/BUILD.bazel:6:12: Compiling Rust bin crate-name (1 files) failed: not all outputs were created or valid

[bitemyapp]

build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cincremental=false
build:linux --@rules_rust//rust/settings:experimental_use_cc_common_link=True
build:linux --linkopt=-fuse-ld=lld

Got it to build with this, but no LTO.

[havasd]

My experience is that, that this error is happening when the local LLVM (lld) version which is used for linking is not compatible with the LLVM version used in the rustc.

That is the indication of the error. However, you said that even using llvm 19 based lld is not working?
Could you please double check the lld version?

[bitemyapp]

My experience is that, that this error is happening when the local LLVM (lld) version which is used for linking is not compatible with the LLVM version used in the rustc.

That is the indication of the error. However, you said that even using llvm 19 based lld is not working? Could you please double check the lld version?

I've uninstalled llvm-18 and it's still happening somehow. It looks like it's trying to use gold from binutils-gold instead of LLVM 19's gold. I'm trying to harmonize Rust 1.86's LLVM 19 with a system-installed LLVM 19.

[bitemyapp]

Potentially relevant thread: rust-lang/rust#49879 (comment)

[bitemyapp]

This combination worked on the Linux CI/CD instance, but I need to make linker-plugin-lto and --linkopt=-fuse-ld=lld linux-only as it failed on my Mac.

build --compilation_mode=opt
build --@rules_rust//rust/settings:lto=thin
build --@rules_rust//rust/settings:extra_rustc_flag=-Cembed-bitcode=true
build --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Ctarget-cpu=native
build --@rules_rust//rust/settings:extra_rustc_flag=-Clinker-plugin-lto
build --linkopt=-fuse-ld=lld

[havasd]

You can use platform-specific config feature in .bazelrc or create your own toolchain.

https://bazel.build/run/bazelrc#enable_platform_specific_config

[bitemyapp]

You can use platform-specific config feature in .bazelrc or create your own toolchain.

https://bazel.build/run/bazelrc#enable_platform_specific_config

Yes, thank you. I already had it I just wasn't leveraging it yet, here's the final version:

common --enable_platform_specific_config
build --compilation_mode=opt
build --@rules_rust//rust/settings:lto=thin
build --@rules_rust//rust/settings:extra_rustc_flag=-Cembed-bitcode=true
build --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Ctarget-cpu=native
build:linux --@rules_rust//rust/settings:extra_rustc_flag=-Clinker-plugin-lto
build:linux --linkopt=-fuse-ld=lld

Hopefully this saves someone else time. I installed llvm-19 and lld-19 on Ubuntu 24.04, uninstalled llvm which was actually llvm-18 as well.

Is there a way for rules_rust to know when to do this automatically? Cargo seems to do this automatically.

[havasd]

I think cargo won't necessarily enable linker-plugin-lto as it is not really needed when linking done by rustc. It doesn't give you too much benefit as far as we found out in our project. Unless you really need cross-language lto