Assertion in on-disk cache code in the compiler fails

[gaultier]

Code

Since this is a caching issue (apparently, since the old code and the new code both build fine on their own) it's hard for me to give code to reproduce it. Basically I had code in a library like this:

// Old code

pub fn foo(u64 x){}

which compiled fine with cargo b --target=x86_64-unknown-linux-gnu.

Then I changed the code to:

// New code

pub fn foo_u64(u64 x){}

pub fn foo_16(u16 x){}

And then re-running cargo b --target=x86_64-unknown-linux-gnu it fails.

Meta

rustc --version --verbose:

rustc 1.76.0 (07dca489a 2024-02-04)

Error output

$ cargo b --target=x86_64-unknown-linux-gnu
   Compiling fmw_rust v0.1.0 (/home/pg/company-code/ [...] /src/src/fmw_rust)
thread 'rustc' panicked at compiler/rustc_middle/src/query/on_disk_cache.rs:525:5:
assertion `left == right` failed
  left: 0
 right: 1002111927320821928687967599834759150
stack backtrace:
   0:     0x7b3863f8b6f6 - std::backtrace_rs::backtrace::libunwind::trace::hbee8a7973eeb6c93
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x7b3863f8b6f6 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8ac75eea3aa6899
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7b3863f8b6f6 - std::sys_common::backtrace::_print_fmt::hc7f3e3b5298b1083
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x7b3863f8b6f6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbb235daedd7c6190
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7b3863fddf40 - core::fmt::rt::Argument::fmt::h76c38a80d925a410
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
   5:     0x7b3863fddf40 - core::fmt::write::h3ed6aeaa977c8e45
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
   6:     0x7b3863f7f53f - std::io::Write::write_fmt::h78b18af5775fedb5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
   7:     0x7b3863f8b4d4 - std::sys_common::backtrace::_print::h5d645a07e0fcfdbb
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7b3863f8b4d4 - std::sys_common::backtrace::print::h85035a511aafe7a8
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7b3863f8e267 - std::panicking::default_hook::{{closure}}::hcce8cea212785a25
  10:     0x7b3863f8dfc9 - std::panicking::default_hook::hf5fcb0f213fe709a
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
  11:     0x7b3866c1461c - std[79729d9c385e1623]::panicking::update_hook::<alloc[6df67106ebca92c0]::boxed::Box<rustc_driver_impl[66ed8fdbde15dc6c]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7b3863f8e9b6 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hbc5ccf4eb663e1e5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
  13:     0x7b3863f8e9b6 - std::panicking::rust_panic_with_hook::h095fccf1dc9379ee
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
  14:     0x7b3863f8e702 - std::panicking::begin_panic_handler::{{closure}}::h032ba12139b353db
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
  15:     0x7b3863f8bbf6 - std::sys_common::backtrace::__rust_end_short_backtrace::h9259bc2ff8fd0f76
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
  16:     0x7b3863f8e460 - rust_begin_unwind
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
  17:     0x7b3863fda645 - core::panicking::panic_fmt::h784f20a50eaab275
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
  18:     0x7b3863fdabdb - core::panicking::assert_failed_inner::hbf94b40c37b92af0
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:342:17
  19:     0x7b3867026247 - core[f975038e3cc9791c]::panicking::assert_failed::<u128, u128>
  20:     0x7b3868e4a531 - <rustc_middle[e0f609162babcb92]::query::on_disk_cache::OnDiskCache>::new
  21:     0x7b3868b4bba2 - rustc_incremental[9a504be5dec41bc5]::persist::load::load_query_result_cache
  22:     0x7b3868b4bdaa - rustc_interface[bf5cafa581ab7832]::passes::create_global_ctxt
  23:     0x7b3868cf2af0 - <rustc_interface[bf5cafa581ab7832]::queries::Queries>::global_ctxt
  24:     0x7b3868afe568 - rustc_interface[bf5cafa581ab7832]::interface::run_compiler::<core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>, rustc_driver_impl[66ed8fdbde15dc6c]::run_compiler::{closure#0}>::{closure#0}
  25:     0x7b3868c2c7db - std[79729d9c385e1623]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[bf5cafa581ab7832]::util::run_in_thread_with_globals<rustc_interface[bf5cafa581ab7832]::interface::run_compiler<core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>, rustc_driver_impl[66ed8fdbde15dc6c]::run_compiler::{closure#0}>::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>
  26:     0x7b3868c2c639 - <<std[79729d9c385e1623]::thread::Builder>::spawn_unchecked_<rustc_interface[bf5cafa581ab7832]::util::run_in_thread_with_globals<rustc_interface[bf5cafa581ab7832]::interface::run_compiler<core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>, rustc_driver_impl[66ed8fdbde15dc6c]::run_compiler::{closure#0}>::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>::{closure#1} as core[f975038e3cc9791c]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  27:     0x7b3863f988e5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h12de4fc57affb195
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  28:     0x7b3863f988e5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3c619f45059d5cf1
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  29:     0x7b3863f988e5 - std::sys::unix::thread::Thread::new::thread_start::hbac657605e4b7389
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  30:     0x7b3863d64b5a - start_thread
                               at ./nptl/pthread_create.c:444:8
  31:     0x7b3863df55fc - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
  32:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.76.0 (07dca489a 2024-02-04) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type staticlib --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `fmw_rust` (lib)

Backtrace

$  RUST_BACKTRACE=1 cargo b --target=x86_64-unknown-linux-gnu
   Compiling fmw_rust v0.1.0 (/home/pg/company-code/ [...] /src/src/fmw_rust)
thread 'rustc' panicked at compiler/rustc_middle/src/query/on_disk_cache.rs:525:5:
assertion `left == right` failed
  left: 0
 right: 1002111927320821928687967599834759150
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:342:17
   3: core::panicking::assert_failed::<u128, u128>
   4: <rustc_middle::query::on_disk_cache::OnDiskCache>::new
   5: rustc_incremental::persist::load::load_query_result_cache
   6: rustc_interface::passes::create_global_ctxt
   7: <rustc_interface::queries::Queries>::global_ctxt
   8: rustc_interface::interface::run_compiler::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#0}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.76.0 (07dca489a 2024-02-04) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type staticlib --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `fmw_rust` **(lib)**

[Noratrieb]

@saethlin

[gaultier]

Confirming that cargo clean and then cargo b --target=x86_64-unknown-linux-gnu makes the problem disappear, so it's probably a caching bug.

Correction: It first works and makes the problem disappear. However, after making a code change (renaming foo_u64 back to foo), and rebuilding, the bug reappears, with a different caching 'tag':

thread 'rustc' panicked at compiler/rustc_middle/src/query/on_disk_cache.rs:525:5:
assertion `left == right` failed
  left: 3733157840821228655755399166
 right: 1002111927320821928687967599834759150

[saethlin]

@Nilstrieb The failing assertion is not the one I recently added. The assertion that's firing here is very old:

rust/compiler/rustc_middle/src/query/on_disk_cache.rs

Line 525 in 07dca48

assert_eq!(actual_tag, expected_tag);

[gaultier]

It happened to me again today, same project, different file, after renaming (through the rust-analyzer language server) a variable.

[saethlin]

As far as I can tell, the ICE reported in this issue is caused by another process damaging rustc's build artifacts or by a buggy filesystem. That other process could be anything from an IDE running the compiler at the same time as a manual build to a find-and-replace that accidentally edits files in the target directory.

#124686 contains a mitigation for this situation; the compiler now has a crude way to detect and ignore damaged incremental compilation files. You will still get a warning about the damaged file, but the build should proceed normally. That change shipped in Rust 1.80 which released today, so I am closing this issue and a few others which as far as I can tell all have the same root cause. You shouldn't see this ICE anymore, but please file a new issue if you see something like it on 1.80 or later.

[gaultier]

Thank you, I have updated to 1.80 and will report back if the issue happens again.

[gaultier]

Epilogue: this issue did not show up again, except once.
What happened is that I renamed a function from “foo” to “bar” with “sd”, the rust command line tool similar to sed.
And then when compiling the issue appeared again.
I am now pretty certain that the root cause is not in the rust compiler code but instead due to this external tool modifying parts of the on disk cache of the compiler (when doing the renaming, since I suppose the old name of the function appears in the on disk cache and it gets renamed there).

If this is a scenario that’s common and folks want to prevent it from happening, I could suggest adding a checksum to the on disk cache to detect silent external modifications?
Or base64 encoding the function names in the on disk cache to prevent them from being detected by such tools as needing to be renamed?